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I.  INTRODUCTION 


A.  Problem  and  Overview 


The  purpose  of  cost-effectiveness  analysis  in  a  Navy  education  and 
training  context  is  to  obtain  information  that  might  be  used  to  choose 
among  alternative  strategies,  methodologies  or  settings  for  achieving  cer¬ 
tain  goals  or  objectives.  Each  alternative  is  analyzed  for  its  actual  or 
predicted  effect  as  in  any  conventional  evaluation  model.  Assumptions  and 
caveats  that  accompany  any  experimental  or  quasi-experimental  design  are 
still  relevant,  but  an  additional  major  component,  cost  analysis,  has  been 
added. 

Little  needs  to  be  said  about  the  existence  of  finite  budgets  and  other 
resources  for  Navy  training  and  accompanying  R&D  activities.  Recent  civilian 
and  military  task  forces  have  commented  about  the  relationships  between 
education  and  training,  the  impact  of  technology,  and  the  requirements  for 
cost-effectiveness  analysis.  In  its  final  summary  report,  the  Task  Force 
on  Training  Technology  for  the  Defense  Science  Board  (1975)  placed  consi¬ 
derable  emphasis  on  the  need  for  cost-effectiveness  analysis  and  devel¬ 
oping  the  capability  to  conduct  such  studies.  Although  Navy  training 
commands,  schools,  or  centers  may  have  relatively  narrow  mission  profiles, 
there  are  presumably  many  feasible  alternative  methods  for  accomplishing 
those  missions.  Cost-effectiveness  analysis  requires  that  alternatives 
be  considered  not  only  on  the  basis  of  mission  accomplishment  (benefit 
or  outcomes)  but  also  the  resources  required  by  each  alternative. 

In  brief,  this  type  of  analysis  is  not  a  new  form  of  evaluation  or 
economic  research  as  much  as  it  is  one  that  combines  cost  considerations 
with  standard  evaluation  designs.  It  is  assumed  that  these  designs  will 
permit  both  ad  hoc  and  predictive  analyses.  A  significant  portion  of 
this  approach  is  devoted  to  effectiveness  or  outcomes  comparison.  These 
data  are  then  combined  with  analyses  of  the  costs  of  implementation  and 
operation  in  order  to  make  cost-effectiveness  comparisons.  These  com¬ 
parisons  will,  then,  theoretically  help  decision-makers  at  many  levels 
select  alternatives  that  will  maximize  desired  outcomes  at  specified 
levels  of  resource  use  or  minimize  costs  for  identified  outcome  levels. 

A  later  section  contains  more  technical  descriptions  of  various  tech¬ 
niques  or  types  of  analyses  that  have  potential  for  aiding  in  this  task, 
but  a  brief  comment  on  terminology  is  in  order.  Evaluators  and  economists 
often  use  the  terms  "effects,"  "outcomes,"  and  "effectiveness"  to  refer 
to  the  definable  results  or  products  of  a  specified  instructional  system. 
Occasionally,  indicators  or  proxy  measures  such  as  the  number  of  students 
processed  are  also  included  under  the  effectiveness  label.  The  term 
"benefit"  is  usually  defined  and  used  as  an  indicator  or  measure  of  the 
/dlue  of  the  above  mentioned  effects.  These  may  include  long-term  indi¬ 
vidual  or  societal  outcomes  as  well  as  system  products.  The  goal  is  to 


generate  a  single  indicator  or  unit  of  value  that  can  then  be  compared 
with  the  dollar  costs  required  to  produce  that  benefit.  This  usually 
results  in  the  translation  of  benefits  to  equivalent  dollar  value  so  that 
both  cost  and  benefit  indices  are  directly  comparable.  More  will  be  said 
about  these  issues  later. 

B.  Purpose  of  Guidelines 

These  condensed  notes  and  guidelines  for  cost-effectiveness  analysis 
are  designed  to  be  used  by  trainers,  educators,  and  researchers  respon¬ 
sible  for  describing  or  comparing  instructional  alternatives.  The  guide¬ 
lines  will  help  make  decisions  about: 

•  What  level  of  expertise  is  needed  to  perform  cost  analyses 

•  What  kind  of  analysis  to  conduct 

•  Which  references  and  examples  to  review  first 

•  What  components  to  include 

•  Which  general  and  specific  criteria  to  use 

•  What  kinds  of  answers  can  be  expected 

•  What  format  to  use  when  reporting 

Although  the  first  sections  review  the  art  and  science  of  systems 
analysis  as  it  is  or  might  be  applied  to  Navy  education  and  training, 
the  focus  of  the  report  is  on  one  subset  of  the  total  process — cost- 
effectiveness  analysis.  To  assume  that  cost-effectiveness  or  any  sub¬ 
system  is  more  important  than  another  is  to  fall  into  the  very  trap  that 
systematic  analysis  is  theoretically  designed  to  prevent,  i.e.,  "we've 
got  the  solution,  now  where's  the  problem?"  Consideration  of  this  kind 
of  analysis  is  important.  A  brief  listing  of  several  of  the  purposes 
it  can  serve  illustrates  the  point.  The  process  may  be  useful  for: 

(1)  determining  the  long-  and  short-term  cost  implications  of  various 
programs,  (2)  predicting  the  initial  investment  and  continuing  operating 
coat  of  an  instructional  innovation,  (3)  evaluating  and  improving  the 
allocation  of  instructional  resources,  (4)  costing  and  testing  the  eco¬ 
nomic  feasibility  of  large-  and  small-scale  plans,  (5)  assessing  ways 
to  improve  training  productivity  and  efficiency,  and  (6)  judging  the 
advantages  and  disadvantages  of  alternative  instructional  strategies. 

This  list  is  not  intended  to  represent  the  total  domain  of  cost-effec¬ 
tiveness  nor  is  it  to  be  assumed  that  the  outcomes  of  such  analyses  or 
the  use  of  such  data  will  always  be  in  the  best  interest  of  trainees. 

When  such  data  are  misused  or  abused,  it  is  usually  the  result  of  naivete 
or  misinformation.  Hopefully,  the  requirement  for  increased  accounta¬ 
bility,  productivity,  and  efficiency  will  also  be  accompanied  by  demands 
to  assess  the  implications  of  alternative  strategies  aimed  at  meeting 
these  requirements.  Cost-effectiveness  is  one  process  that  may  allow 
collection,  categorization,  and  assessment  of  the  data  needed  to  analyze 
those  alternatives. 
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Scope 


The  following  section  relates  the  concepts  and  methodology  of  cost- 
effectiveness  analysis  to  systems  analysis  terminology  and  modeling. 

The  third  section  identifies  a  set  of  design  considerations  that  must 
be  resolved  before  any  analysis,  particularly  cost-effectiveness  analysis, 
can  be  conducted. 

Sections  four  and  five  contain  the  primary  ingredients  of  this 
document,  the  guidelines  and  recommendations  for  designing  a  cost-effec¬ 
tiveness  analysis  study.  Section  four  is  organized  around  a  series  of 
interrelated  subsets  of  an  analysis  and  includes  accompanying  descriptions 
and  taxonomies  to  assist  a  designer.  Section  five  is  an  expansion  of 
one  subset  in  section  four.  More  extensive  cost  categorization  consid¬ 
erations,  cost  model  format  and  components,  and  a  summary  cost  matrix 
are  included. 

An  effort  was  made  to  limit  the  number  and  maintain  the  quality  of 
references.  A  selected  annotated  bibliography  is  included  for  those 
desiring  further  clarification  or  examples.  References  in  this  section 
have  been  selected  for  maximum  utility  to  Navy  education  and  training 
analysts. 

Appended  are  several  documents.  Appendices  A  and  B  describe  the  allo¬ 
cation  of  instruction  costs,  for  facilities  and  personnel,  respectively. 
Appendix  C  presents  an  actual  executive  level  cost-effectiveness  analysis 
report  (U.  S.  Army  Training  Extension  Course  Cost-Effectiveness  Study), 
and  Appendix  D  summarizes  a  preliminary  cost  analysis  of  a  Navy  training 
course.  Finally,  Appendix  E  contains  two  prototype  cost  data  collection 
instruments  for  use  in  obtaining  costs  for  Navy  training  courses  or  pro¬ 
grams. 


Neither  the  guidelines  nor  the  examples  are  intended  to  be  used  as 
blueprints  since  each  setting  will  require  potentially  different  resource 
mixes,  instructional  strategies,  evaluative  criteria,  and  the  like. 
Planners  and  analysts  conducting  such  studies  should,  however,  be  able 
to  use  these  to  design  a  comprehensive  cost-effectiveness  study  that 
includes  useful  cost  analysis  categories  and  summaries  as  well  as  appro¬ 
priate  comparative  criteria. 


II.  SYSTEMS  ANALYSIS:  A  CONTEXT  FOR  COST-EFFECTIVENESS  ANALYSIS 


A.  Discussion  of  Systems  Analysis 

As  Navy  training  and  education  have  grown  both  in  scope  and  complexity, 
those  responsible  have  become  increasingly  less  secure  in  the  decision¬ 
making  process.  Difficulties  in  predicting  the  effect  of  a  particular 
alternative  on  the  whole  system  of  instruction  is  multiplying  exponen¬ 
tially  from  year  to  year.  Accountability  requirements,  tight  money, 
and  second-guess  detractors  continuously  exert  an  inhibiting  pressure 
on  serious  attempts  to  implement  quality  instruction,  particularly  when 
the  attempts  are  innovative.  In  such  contexts,  systems  analysis  provides 
a  method  for  identifying  and  clarifying  the  important  elements  in  the 
instruction-learning  process.  Used  appropriately,  it  presents  the  various 
components  of  the  total  instructional  process,  helps  to  describe  the 
relationships  between  the  components,  and  assists  with  assessing  their 
influence  on  one  another.  The  analyst  can  aggregate  isolated  constituents 
into  an  interacting  whole  and  can  resolve  the  anonymity  of  these  elements 
into  a  dynamic  process.  This  is  not  only  a  strong  methodological  approach 
for  the  unification  of  knowledge,  but  a  facilitator  of  communication. 

This  section  places  cost  analysis  concepts  into  the  context  of  the 
more  comprehensive  systems-analysis  framework.  This  overview  of  systems 
analysis  discusses  both  its  principles  and  process.  The  principles 
emphasize  the  exigency  of  model  construction,  and  the  analysis  process 
employs  a  taxonomic  classification  to  describe  concepts,  purposes,  and 
approaches.  This  brief  explanation  of  relevant  concepts  provides  a  frame¬ 
work  for  interpreting  the  remaining  sections. 

Analysis  of  instructional  alternatives  is  the  focus  of  the  power 
of  training  systems  analysis.  It  is  from  results  of  careful  systematic 
analyses  that  the  alternatives  indicating  the  greatest  promise  can  be 
identified.  Results  of  existing  approaches  in  Navy  training  and  the 
potential  of  new  techniques  can  be  compared,  priorities  can  be  set,  and 
decisions  can  be  made  on  the  basis  of  comparative  data  from  the  analysis. 

Before  discussing  possible  approaches  to  systems  analysis  in  educa¬ 
tion,  a  clarification  of  existing  terminology  may  be  helpful.  The  literature 
on  systems  analysis  is  quite  extensive  and  redundant.  A  serious  inves¬ 
tigator  might  consult  one  or  more  of  the  sources  described  in  the  annotated 
bibliography.  Each  of  those  authors  has  contributed  significantly  to 
the  state-of-the-art  as  it  now  exists.  Two  additional  general  references — 
Introduction  to  Operations  Research  (Churchman,  Ackoff,  and  Arnoff, 

1957)  and  Educational  Systems  Analysis  (Banghart,  1969) — are  particularly 
useful  initial  texts. 

The  term  "systems  analysis"  coexists  with  many  similar  terms — opera¬ 
tions  research,  economic  analysis,  cost-benefit  analysis,  etc.  The 
important  characteristic  they  all  have  in  common  is  the  emphasis  on  making 
comparisons  systematically  in  quantitative  and  qualitative  terms  using 
logical  sequences  that  can  be  retraced  and  verified.  It  is  always  with 
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this  paramount  characteristic  that  "systems  analysis"  is  used  in  this 
text.  Kaufman's  (1972)  comments  about  the  communication  problems  caused 
by  the  discrepancies  in  definition  are  worth  reading.  His  definitions 
of  "system,"  "system  analysis,"  and  "system  approach"  serve  as  a  useful 
guide  for  all  analysts.  A  lively  discussion  of  the  semantics  issue  may 
be  found  in  Quade's  Analysis  for  Military  Decisions,  1964,  pp.  2-12. 

The  term  "analyst"  denotes  an  individual  or  team  of  individuals  in¬ 
terested  in  the  process  of  systematically  examining  objectives,  policies, 
strategies,  costs,  effectiveness,  and  feasibilities  of  training  and  educa¬ 
tion.  Degree  of  proficiency  aside,  "analysts"  may  be  evaluators,  research¬ 
ers,  developers,  or  instructors,  but  application  of  the  process  of  systems 
analysis  also  makes  them  analysts. 

B.  Systems  Analysis  Modeling 

In  this  context,  the  goal  of  systems  analysis  is  to  aid  In  the  formula¬ 
tion,  analysis,  and  choice  of  future  courses  of  action.  Systematic  con¬ 
sideration  of  any  problem  requires  that  some  model  be  used.  Depending 
upon  your  concept  of  "model,"  it  may  be  unstated  and  undeveloped,  but 
if  a  decision  maker  does  something  other  than  toss  a  coin  in  the  air, 
a  model  of  some  type  is  being  employed. 

The  assumption  that  humans  base  decisions  on  some  type  of  mental 
model  is  an  important  factor  in  the  rationale  underlying  systems  analysis. 
If  humans  require  a  model,  albeit  only  an  intuitive  one,  then  the  model 
providing  the  best  information  should  lead  to  better  decision  making. 

The  better  model  is  determined  by  its  ability  to  provide  a  better  descrip¬ 
tion,  explanation,  or  prediction.  This,  of  course,  must  always  be  tem¬ 
pered  by  considerations  of  complexity  and  utility. 

1.  Analysis  Process 

It  is  important  to  emphasize  that  an  analysis  model  is  not  a 
flowchart,  but  a  process.  Flowcharts,  descriptions,  and  examples  only 
serve  to  communicate  the  purpose  and  practice  of  the  process. 

There  are  as  many  approaches  to  systems  analysis  modeling  as 
there  are  systems  analysts.  Churchman,  Ackoff,  and  Arnoff  (1957),  Quade 
and  Boucher  (1968),  Seiler  (1969),  and  Fisher  (1971)  provide  thorough 
model  design  guidelines  although  none  are  specifically  intended  for  Navy 
training  application. 

The  systems  analysis  process  involves  a  series  of  iterations 
through  several  phases.  Quade  and  Boucher  (1968)  describe  these  phases 
as  formulation,  search,  evaluation,  interpretation,  and  verification. 

a.  Formulation.  The  formulation  (conceptual)  phase  involves 
the  definition  of  the  problem,  clarification  of  objectives,  and  the 
establishment  of  evaluative  criteria.  Examples  of  techniques  or  procedures 
included  within  this  phase  Include:  front  end  analysis,  needs  assessment, 
values  clarification,  priority  setting,  and  goal  analysis.  Common  pitfalls 
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in  this  domain  include:  failing  to  allocate  and  spend  enough  of  the 
total  time  available  for  a  study  deciding  what  the  problem  really  is, 
determining  objectives  and  criteria  carelessly,  and  trying  to  do  too 
big  a  job. 


b.  Search.  The  search  (research)  phase  includes  the  collection 
of  data,  the  establishment  of  predictive  relationships,  and  the  generation 
of  alternative  strategies  that  might  have  some  chance  of  solving  the 
problem.  A  common  pitfall  associated  with  this  phase  relates  to  an 
analyst  examining  an  unduly  restricted  range  of  alternatives. 

c.  Evaluation.  The  evaluation  (analytic)  phase  involves  the 
building  of  various  models,  the  use  of  those  models  to  predict  the  con¬ 
sequences  of  the  selection  of  each  alternative,  and  the  comparison  of 
the  alternatives  in  terms  of  those  consequences.  Common  pitfalls  here 
include:  forcing  a  complex  problem  into  an  analytically  tractable  frame¬ 
work  by  overemphasizing  ease  of  computation,  becoming  more  interested 

in  the  details  of  the  model  than  in  the  real  world,  using  improper  costing 
concepts,  and  trying  to  do  too  big  a  job. 

d.  Interpretation.  The  interpretation  (judgmental)  phase  in¬ 
volves  the  use  of  predictions  obtained  from  the  models  and  any  relevant 
information  to  compare  the  alternatives,  to  derive  conclusions  from  them, 
and  to  suggest  a  course  of  action.  A  common  pitfall  here  is  to  become  more 
interested  in  the  details  of  the  model  than  in  the  real  world. 

e.  Verification.  The  verification  (validation)  phase  involves 
testing  the  assumptions,  conclusions,  and  recommendations  against  some 
set  of  criteria.  One  common  pitfall  here,  particularly  in  large-scale 
projects,  is  a  failure  to  take  proper  account  oc  various  uncertainties 
such  as  technical  or  theoretical  advancements  i*  he  state-of-the-art, 
or  large  trainee  demand  fluctuations. 

Although  all  of  the  phases  are  critical,  suade  (1964)  recommends 
particular  emphasis  be  placed  on  the  formulation  phase: 

It  is  difficult  to  overemphasize  the  importance  of  a 
careful  formulation.  It  should  identify  the  subproblems 
involved,  isolate  the  major  factors,  develop  a  vocabulary 
for  dealing  with  them,  stretch  out  the  relationships  be¬ 
tween  the  variables  as  they  appear,  and  even  arrive  at 
a  tentative  set  of  conclusions.  The  idea  is  to  make 
clear  the  structure  of  the  analysis,  but  more  importantly, 
it  offers  a  concrete  hypothesis  for  others  to  probe. 

(p.  307) 

An  excellent  discussion  of  the  problems  and  pitfalls  associated 
with  the  design  of  such  studies  is  provided  by  Kazanowski  in  his  chapter 
entitled,  "Cost-effectiveness  Fallacies  and  Misconceptions  Revisited" 
(English,  1968,  p.  151-165). 
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2.  Model  Build In 


Fisher  (1964)  comments  that  a  model,  depending  on  the  nature 
of  the  problem,  can  be  formal  or  informal,  highly  mathematical  or  not 
at  all  mathematical,  computerized  or  manual.  He  emphasizes  that  a  model 
need  not  be  highly  sophisticated  or  formal  to  be  useful.  He  lists 
several  points  that  a  model  builder  should  keep  in  mind: 

•  Model  building  is  an  art,  not  a  science.  It  is  often  an  exper¬ 
imental  process. 

•  The  major  function  of  the  model  is  the  inclusion  of  those 
factors  which  are  relevant  to  the  problem  at  hand,  and  the  judicious 
suppression  of  those  which  are  relatively  unimportant.  Unless  the  latter 
is  done,  the  model  is  likely  to  be  unmanageable. 

•  The  main  purpose  in  designing  the  model  is  to  develop  a  meaning¬ 
ful  set  of  relationships  among  objectives,  the  relevant  alternatives 
available  for  attaining  the  objectives,  the  estimated  cost  of  the  alter¬ 
natives,  and  the  estimated  utility  for  each  of  the  alternatives. 

•  Provision  must  be  made  for  explicit  treatment  of  uncertainty. 

•  Since  by  definition  a  model  is  an  abstraction  from  reality, 
the  model  must  be  built  on  a  set  of  assumptions.  These  assumptions 
must  be  made  explicit.  If  they  are  not,  this  is  to  be  regarded  as  a 
defect  of  the  model  design  (Fisher,  1964,  p.  10). 

a.  Model  Types.  Following  Fisher's  guidelines,  the  analyst 
prepares  to  select  the  model.  The  purpose  of  the  analysis,  the  kinds 
of  established  criteria,  and  types  of  data  available  influence  the  ana¬ 
lyst's  final  selection.  Two  models  appropriate  for  education  and  training 
are  mathematical  models  and  tabular  display  (matrix)  models. 

Mathematical  models  are  suitable  when  all  the  evaluative 
criteria  can  be  quantified  and  are  comparable.  For  most  Navy  training 
and  education  contexts,  this  is  not  feasible. 

A  tabular  display  model  format,  on  the  other  hand,  allows 
more  flexibility.  Using  a  variety  of  measures,  the  analyst  can  compare 
alternatives.  This  has  the  decisive  advantage  of  allowing  uncomparable 
common  measures  to  be  used  as  descriptors  of  alternatives.  In  the  tabular 
display  model,  the  analyst  can,  for  instance,  incorporate  qualitative 
statements  on  relative  effectiveness  and  efficiency.  Unquantif lable 
measures  are  often  times  heavily  value-laden  and  demand  special  considera¬ 
tion.  This  additional  factor  almost  requires  a  tabular  display  model 
in  most  instructional  systems  analysis  comparisons.  Such  articulation 
of  values,  priorities,  and  other  various  publicly  stated  evaluative  cri¬ 
teria  can  be  incorporated  into  a  tabular  array.  Examples  of  tabular 
arrays  are  included  in  later  sections  of  this  review. 


b.  Uncertainty.  Predictive  analyses  of  instructional  alternatives 
include  many  elements  of  uncertainty.  These  include  uncertainty  about 

the  state  of  the  world.  Navy  school-based  training,  or  shipboard  training, 
as  well  as  uncertainty  about  change  elements  that  might  exist  even  if 
environmental  uncertainties  did  not  exist.  Quade  (1963),  Fisher  (1964), 
and  Hitch  (1960)  have  written  extensively  on  the  topic  as  it  relates  to 
national  security  analysis.  Much  of  this  can  be  applied,  with  modifi¬ 
cation,  to  Navy  training  analysis. 

A  technique  for  treating  uncertainty  that  appears  to  have 
general  promise  is  sensitivity  analysis.  For  variables  about  whicli  the 
analyst  is  very  uncertain,  application  of  this  technique  will  at  least 
provide  several  estimated  values  rather  than  one  expected  value.  A  range 
of  values  for  a  variable  may  be  plugged  into  a  model  to  see  what  effect 
each  will  have  on  the  outcome.  If  there  are  many  of  these  variables, 
considerable  calculation  is  required  and  the  question  then  arises  as 
to  the  value  of  the  exercise  in  relation  to  the  usefulness  of  the  infor¬ 
mation  generated. 

c.  Validation.  Accuracy  of  prediction  is  an  indicator  of  the 
validity  of  a  predictive  model.  It  is  difficult  to  validate  a  model 
without  actually  developing  and  testing  generated  alternatives.  Cost 
and  time  constraints  usually  prohibit  this  type  of  testing.  This  is  not 
an  uncommon  circumstance.  Quade  (1963,  p.  20)  suggests  several  criteria 
that  can  be  used  to  check  the  validity  of  many  types  of  models.  He  asks: 

•  Can  the  model  describe  known  facts  and  situations  reason¬ 
ably  well? 

•  When  the  principal  parameters  involved  are  varied,  do 
the  results  remain  consistent  and  plausible? 

•  Can  it  handle  special  cases  for  which  we  already  have  some 
indication  as  to  what  the  outcome  should  be? 

•  Can  it  assign  causes  to  known  effects? 

Each  of  these  issues  must  be  kept  in  mind  when  actually  con¬ 
ducting  a  study.  The  following  section,  however,  discusses  several  issues 
that  must  be  resolved  even  before  any  analysis  is  begun.  The  apparently 
illogical  sequence  is  intentional  since  an  understanding  of  the  context 
is  important  for  judging  the  relevance  of  later  sections. 


III.  PRELIMINARY  CONCERNS 


Before  conceptualizing  a  model,  the  Navy  planner  must  have  a  thorough 
understanding  of  the  concepts,  purposes,  and  alternative  approaches  to 
systems  analysis. 

This  section  describes  five  concerns  that  require  careful  consideration. 
These  five  are  interrelated,  but  relatively  exclusive.  Each  is  defined 
and  broken  into  subcategories.  Whether  the  focus  of  an  analysis  in  a 
Navy  training  or  education  setting  is  on  an  entire  program,  a  unit  of 
instruction,  or  a  strategy,  the  following  general  concerns  must  be  well 
defined: 

•  Level  of  decision  making 

•  Function  of  decision  makers 

•  Purpose  of  cost-effectiveness  analysis 

•  Approach  to  cost-effectiveness  analysis 

•  Type  of  analysis 

Subsequent  pages  contain  brief  definitions  of  these  concepts,  addi¬ 
tional  subcategories,  and  initial  guidelines  for  their  use. 

A.  Level  of  Decision  Making 

One  of  the  first  determinations  an  analyst  must  make  is  the  decision¬ 
making  level  the  model  must  address.  If  the  model  is  to  be  used  only 
by  top  level  commanders  and  program  directors,  the  detail  of  the  model 
need  not  be  extensive.  If  the  data  are  to  be  used  by  project  managers 
and  instructional  designers,  the  information  level  reaches  substantially 
below  the  program  level,  perhaps  to  the  level  of  a  unit  of  instruction 
or  even  the  performance  objective  level. 

It  requires  considerably  more  time  and  resources  to  collect  and 
analyze  data  at  these  lower  levels.  If  there  is  no  need  for  detailed 
cost  effectiveness  and  feasibility  data,  the  analyst  needs  only  to  as¬ 
semble  information  at  the  appropriate  higher  level  of  interest.  The 
key  is  to  determine  the  appropriate  level  or  levels  of  interest,  thus 
establishing  the  specificity  of  information  the  model  must  provide. 

The  analyst  focusing  on  larger  issues  must  identify  the  decision 
makers  who  have  the  authority  to  initiate,  terminate,  and  alter  policies 
governing  the  alternatives  being  analyzed.  It  is  important  to  know  how 
these  people  made  decisions  so  that  the  results  of  the  analysis  can  be 
properly  structured  and  presented. 

Consideration  should  be  given  to  the  the  following  questions  which 
assist  in  the  determination  of  specificity: 

•  Who  will  be  using  the  information  furnished  by  the  model? 

•  Who  is  to  be  implementing  the  model? 
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•  How  is  It  to  be  implemented  (hand  calculations  or  sophisticated 
electronic  hardware)? 

•  What  is  the  complexity  of  that  being  represented  by  the  model? 

»  How  often  is  the  model  to  be  used? 

•  What  degree  of  precision  does  the  model  need  to  have  in  relation¬ 
ship  to  the  real  world? 

•  How  accurate  are  the  data  that  will  be  used  in  the  model? 

•  How  accurate  do  the  data  need  to  be? 


B.  Function  of  Decision  Makers 

There  are  several  decision-oriented  activities  to  which  systems  ana¬ 
lysis  contributes.  Each  has  a  specific  purpose  and  an  identifiable  out¬ 
come.  Each  may  be  closely  dependent  upon  another.  Four  important  such 
functions  are  control,  planning,  evaluation,  and  development. 


1.  Control 

The  educational  manager,  chief,  trainer,  or  instructor  must 
be  able  to  effectively  manage  the  resources  under  his  responsibility. 

This  management  cannot  be  done  after  the  vouchers  or  purchase  orders 
are  signed.  Cost  control  practices  in  particular  can  only  be  implemented 
before  the  fact.  However,  after-the-fact  cost  and  effectiveness  account¬ 
ing  information  will  help  decision  makers  control  ongoing  and  future 
programs . 


Planning 


Cost  analysis  information  allows  planning  for  future  changes 
in  allocation  of  resources  for  a  course  or  a  program.  This  Information 
can  allow  for  planning  of  potential  expanded  uses  such  as  extension 
courses,  shipboard  instruction,  or  joint  service  application.  It  can 
also  provide  general  information  for  use  in  planning  adaptations  of 
successful  techniques  or  strategies  to  other  suitable  learning  situations. 
General  analyses  can  also  produce  a  range  of  potential  courses  of  action 
by  means  of  a  systematic  study  of  feasible  alternatives. 


Oftentimes,  planners  are  concerned  with  optimizing  their  own 
programs  at  the  expense  (or  exclusion)  of  other  programs.  Obviously, 
proper  planning  will  not  eliminate  this  "optimization  of  subsystems" 
phenomenon!  but  it  might  help  broaden  some  perspectives. 


3.  Evaluation 

After-the-fact  cost  and  effectiveness  analysis  data  are  valuable 
for  use  in  comparison  with  planned  costs  and  outcomes.  This  pre-post 
comparison  yields  better  planning  data  for  decisions  about  the  value 
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or  utility  of  revisions  or  new  projects.  After-the-fact  analysis  in¬ 
formation  provides  the  decision  maker  with  another  basis  for  judging 
the  success  or  failure  of  a  program.  The  result  may  be  greater  instruc¬ 
tional  effectiveness  or  more  efficient  research  and  development  efforts. 

4.  Development 

Cost  and  effectiveness  analysis  information  can  assist  in  more 
efficient  and  effective  development  of  materials  and  procedures  for  in¬ 
struction.  Such  information  can  aid  in  the  design  of  better  plans  and 
better  management  strategies,  and  in  the  development  of  better  techniques 
for  producing,  validating,  and  reproducing  instructional  materials. 

C.  Purpose  of  Cost-Effectiveness  Analysis 

In  addition  to  levels  and  functions,  three  general  purposes  of  systems 
analysis — description,  comparison,  and  prediction — must  be  considered. 

They  provide  guidelines  for  the  design  of  all  analyses,  including  com¬ 
ponents  such  as  data  collection  instruments  and  procedures  and  report 
specifications.  Figure  1  presents  a  graphic  display  of  these  relation¬ 
ships. 


1.  Descriptive  Analysis 

Descriptive  analysis  is  employed  to  describe  ongoing  or  completed 
programs.  Accounting  and  assessment  data  acquired  in  the  application 
of  this  type  of  analysis  provide  the  basis  for  all  other  analyses.  Cost 
analysis  (accounting)  is  used  to  report  dollar  costs  and,  where  possible, 
to  relate  costs  to  output.  Effectiveness  (or  benefit)  analysis,  for 
example,  may  be  conducted  by  measuring  cognitive  and  noncognitive  changes 
in  learners  (output)  in  relation  to  stated  objectives. 

2.  Comparative  Analysis 

As  stated  earlier,  the  ultimate  goal  of  systems  analysis  is  to 
make  comparisons.  Comparative  analyses  are  based  on  both  descriptive 
and  predictive  analysis.  Ex  post  facto  comparisons  can  be  made  by  using 
data  from  properly  designed  descriptive  analyses.  A  priori  comparisons 
make  use  of  predictive  analyses  which  in  turn  are  built  upon  descriptive 
studies.  For  this  purpose,  distinctions  must  be  made  between  the  various 
types  and  approaches. 

3.  Predictive  Analysis 

Predictive  analysis  is  used  by  those  who  wish  to  ascertain  costs 
and  benefits  of  proposed  alternatives.  This  requires  consideration  of 
prior  costs  as  a  basis  for  a  predictive  cost  analysis.  It  also  neces¬ 
sitates  determination  of  objectives,  strategies,  required  resources, 
cost  estimating  relationships,  and  the  costing  of  resources.  Effective¬ 
ness  analysis  is  used  to  predict  the  level  (s)  of  output  or  effectiveness 
of  each  alternative.  Predicting  various  levels  of  effectiveness  based 
on  previous  studies  is  one  of  the  most  difficult  and  least  precise 
activities  a  systems  analyst  can  perform. 
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CLASSIFICATION  SYSTEM  OF  COSTING  TECHNIQUES 


Figure  1.  Classification  System  of  Costing  Techniques. 


The  inexact  nature  of  such  activities  oftentimes  leads  to  condemns 
tion  of  the  entire  process;  but  systems  analysis  is  primarily  intended 
to  reduce,  not  eliminate,  the  margin  of  error  in  the  decision-making 
process. 

Several  additional  subsets  of  these  purposes  can  be  easily  iden¬ 
tified.  Examples  include:  (a)  allocating  resources  among  identified 
objectives,  (b)  choosing  alternative  means  to  meet  the  given  objectives, 
(c)  assessing  the  worth  of  different  objectives,  and  (d)  providing  for 
the  systematic  generation  of  alternatives  which  were  not  originally 
identified. 

D.  Approach  to  Cost-Effectiveness  Analysis 


In  comparing  alternatives  in  terms  of  the  data  generated  by  the 
analysis  model,  two  basic  conceptual  approaches — fixed  effectiveness 
and  fixed  cost — may  be  used. 

In  the  fixed-effectiveness  approach,  a  desired  level  of  effectiveness 
or  output  is  specified  and  then  the  cost  of  alternative  means  of  achieving 
that  specified  level  is  examined.  The  intent  is  to  identify  the  alterna¬ 
tive  or  combination  of  alternatives  which  will  reach  that  level  with 
the  lowest  expenditure  of  resources.  This  is  appropriate  if  defensible, 
comprehensive  indicators  of  effectiveness  can  be  identified.  Most 
instructional  alternatives  bring  about  learner  change  of  several  kinds 
and  to  different  degrees  so  identification  of  such  measures  is  one  of 
the  most  difficult  of  the  analytical  tasks. 

In  the  fixed-cost  approach,  a  single  budget  level  (cost)  is  fixed 
and  then  the  level (s)  of  effectiveness  that  might  be  achieved  through 
different  alternatives  is  examined.  The  intent  is  to  identify  the  alter¬ 
native  or  combination  of  alternatives  which  will  produce  the  highest 
effectiveness. 

Carpenter  and  Haggart  (1970)  recommend  the  fixed-cost  approach  because 
it  focuses  the  attention  of  decision  maker  and  analyst  on  the  effective¬ 
ness  of  alternatives  which  is  by  far  the  most  difficult  phase  of  the 
analysis  process.  The  problems  associated  with  comparing  effectiveness 
of  instructional  alternatives  would  indicate  that  the  fixed-cost  approach 
is,  usually,  more  suitable  for  Navy  training  purposes.  Costs  are  usually 
fixed  by  adjusting  the  dimensions  of  each  alternative  (such  as  the  number 
of  trainees  enrolled)  so  that  each  alternative  incurs  approximately  the 
same  total  cost  over  a  common  time  period. 

A  third  hypothetical  alternative  allows  both  the  cost  and  effective¬ 
ness  to  vary.  This  permits  the  naive  idealist  to  request,  "I  would  like 
to  get  the  maximum  possible  effectiveness  for  the  least  possible  cost." 
Carried  to  its  logical  conclusion,  the  approach  would  result  in  infin¬ 
ite  effectiveness  for  zero  cost.  However,  this  usually  reverts  back  to 
the  fixed-cost  approach,  where,  at  some  stage,  limits  are  placed  on  the 
amount  of  resources  available. 


A  variation  of  these  two  approaches  is  possible  if  alternatives 
differ  in  terms  of  their  relative  effectiveness  and  respective  costs 
on  some  common  evaluative  criterion  that  can  serve  as  the  basis  for  a 
unit  cost  comparison.  For  instance,  where  one  approach  results  in  70 
percent  on  the  criterion  for  cost  X,  and  another  results  in  85  percent 
on  criterion  for  cost  Y.  Where  this  occurs,  disparate  alternatives  can 
still  be  compared  by  using  two  common  variables  such  as  total  system 
(life-cycle)  costs  and  the  effectiveness  measures. 

In  this  case,  however,  comparisons  between  alternatives  must  be  made 
upon  a  feasible  range  of  activity  since  the  unit  costs  for  each  will 
be  likely  to  change  as  the  scale  or  quantity  of  activity  changes.  An 
example  of  this  would  occur  when  a  labor-intensive  instructional  alter¬ 
native  is  compared  to  a  "high  technology"  option.  At  relatively  low 
levels  of  learner  throughput,  the  low  front-end  costs  of  the  labor-inten¬ 
sive  option  would  make  this  one  most  attractive.  At  higher  levels  of 
activity,  however,  the  economy  of  scale  would  begin  to  give  the  advantages 
to  the  high  technology  (higher  initial  but  lower  operating  cost)  alter¬ 
native.  An  example  of  this  kind  of  comparison  can  be  found  in  the  post¬ 
secondary  education  study  reported  by  Doughty  and  Stakenas  (1973). 

E.  Type  of  Analysis 

To  complete  this  section  on  conceptualization,  six  types  of  analysis 
of  training  and  education  programs  in  descending  order  of  sophistication 
are  described:  (1)  cost  benefit,  (2)  cost-effectiveness,  (3)  unit-cost, 
(4)  cardinal  weighting,  (5)  ordinal  ranking,  and  (6)  best-guess  decision 
making.  Some  mention  of  analytical  approach  is  included  where  appropri¬ 
ate. 

1.  Cost-Benefit  Analysis 

Cost-benefit  analysis  is  an  analytical  approach  to  the  selection 
of  alternatives  on  the  basis  of  comparisons  of  benefits  and  costs.  This 
option  is  appropriate  only  when  alternative  outputs  can  be  quantified 
in  dollar  terms,  thereby  heavily  incorporating  judgments  of  value  into 
the  analysis.  Estimates  of  social  value  of  the  output (s)  of  each  alter¬ 
native  are  given  dollar  values  so  that  the  benefits  of  each  alternative 
can  be  expressed  as  a  single  measure — dollars.  This  is  the  only  approach 
which  will  permit  direct  comparisons  of  alternatives  with  different  objec¬ 
tives.  By  translating  the  output  (s)  of  each  alternative  to  dollar  terms, 
the  common  comparisons  are  possible. 


A  benefit-cost  ratio  for  each  alternative  can  be  derived  by  di¬ 
viding  the  value  of  the  benefits  by  the  value  of  the  costs.  Once  the 
costs  and  benefits  are  determined  and  computed,  most  authors  indicate 
that  total  benefits  should  be  greater  than  the  total  costs  (B-C  ratio 
greater  than  1.0)  if  a  program  or  alternative  is  to  be  considered. 
Computations  must  be  made  and  a  benefit-cost  ratio  derived  for  each 
alternative  before  comparisons  can  be  made.  Results  or  changes  that 


are  not  related  to  specified  objectives  are  not  considered  benefits. 

Many  of  these  by-products  or  spillovers  can  be  identified  and  analyzed 
but  only  as  supplementary  gains,  not  as  benefits. 

McKean  (1958,  pp.  34-37,  107-113)  and  Hitch  and  McKean  (1960, 
pp.  166-167)  have  written  lucid  and  enlightening  discussions  of  the 
possible  treacherous  nature  of  ratios.  Eckstein  (1958),  on  the  other 
hand,  bases  many  of  his  analyses  on  the  development  and  use  of  ratios. 
Dorfman's  (1963)  analysis  is  perhaps  most  insightful:  "The  heart  of 
the  matter  lies  in  deciding  what  benefits  should  be  included  and  how 
they  should  be  valued.  The  debate  about  benefit-cost  analysis  centers 
on  the  question  of  whether  the  social  value  of  benefits  can  be  estimated 
reliably  enough  to  justify  the  trouble  and  effort  involved  in  a  benefit- 
cost  computation"  (p.  8). 

New  statistical  models  and  techniques  hold  great  promise  as  solu¬ 
tions  to  some  of  the  problems  involved  in  calculating  benefits  in  dollar 
terms.  Additional  research  and  development  will  help  as  the  art  becomes 
more  scientific  and  better  data  are  obtained. 

2.  Cost-Ef fectivencss  Analysis 

Cost-effectiveness  analysis  is  appropriate  when  a  market  evalua¬ 
tion  or  value  cannot  or  should  not  be  placed  on  the  outputs  of  the  alter¬ 
natives,  but  the  resources  (inputs)  of  the  alternatives  can  be  evaluated 
or  measured  in  market  prices  (dollars).  The  approach  is  derived  from 
contexts  where  the  problem  is  to  select  a  strategy  or  product  from  a 
set  of  alternatives  designed  to  achieve  specified  objectives. 

As  mentioned  in  the  previous  section,  the  recommended  approach 
is  to  construct  fixed  or  equal-cost  alternatives  so  that  each  will  require 
approximately  the  same  total  cost  for  a  fixed  time.  Projection  of  es¬ 
timated  costs  of  each  alternative  can  be  done  with  more  confidence  than 
projection  of  effectiveness,  particularly  since  no  single  measure  of 
effectiveness  will  likely  reflect  the  total  output  of  an  instructional 
alternative. 

Cost-effectiveness  analysis  is  most  useful  for  resolving  problems 
that  hinge  on  allocation  of  resources  because  the  techniques  and  systems 
required  for  the  analysis  of  costs  are  somewhat  more  sophisticated  and 
better  developed  than  those  for  the  analysis  of  effectiveness.  Stated 
differently,  it  is  usually  easier  to  specify  the  cost  variables  in  an 
educational  system  than  it  is  to  determine  the  larger  set  of  variables 
needed  to  account  for  effectiveness.  This  is  not  to  say  that  precise 
accounting  or  prediction  of  such  costs  is  easy.  It  should  also  be  empha¬ 
sized  that  the  cost-effectiveness  approach  will  provide  little  help  in 
determining  whether  an  alternative  or  an  objective  is  worthwhile.  Value 
judgments  of  this  sort  are  left  to  decision  makers  and  are  not  generally 
identified  and  quantified  as  in  the  cost  benefit  approach.  However, 
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many  of  Che  evolving  needs  assessment  and  futures  generating  processes 
(e.g.,  Delphi)  have  potential  for  assisting  In  this  "data  gathering  for 
value  decisions"  area. 


3.  Unit-Cost  Analysis 

Unit-cost  analysis  helps  a  decision  maker  see  what  components 
comprise  the  cost  for  a  course,  alternative,  or  a  strategy  and  how  changes 
in  these  components  will  affect  cost  levels.  Unit  costs  allow  a  decision 
maker  to  predict  future  costs  when  changes  are  introduced  in  instructor 
salary  or  rank,  instructor  work  load,  enrollment,  or  other  variables. 

Quantitative  and  qualitative  effects  of  changes  in  alternatives 
are  not  usually  measured  or  reflected  in  unit  costs.  Most  unit-cost 
analyses  are  based  on  jurisdictional  costs  (usually  derived  from  line-item 
budgets)  which  do  not  reflect  the  total  costs  required  to  realize  some 
specified  outcome.  They  are  far  too  general  to  be  used  as  a  basis  for 
cost-benefit  or  cost-effectiveness  analysis.  Those  which  do  reportedly 
use  functional  costs  (related  to  output)  are  able  to  provide  the  kinds 
of  information  that  can  better  aid  the  decision-making  process. 

Care  must  be  taken  to  avoid  misconceptions  that  often  result 
from  the  use  of  unit-cost  analysis  data.  Use  of  unit  cost  alone  empha¬ 
sizes  the  fiscal  aspect  to  the  exclusion  of  the  effectiveness  of  an 
alternative.  There  is  a  greater  temptation  on  the  part  of  decision  makers 
to  create  more  "efficiency"  through  cost  reductions  than  to  improve  the 
effectiveness.  It  is  also  easy  to  make  improper  comparisons  between 
courses,  curricula,  or  departments  which  are  not  comparable  in  nature. 

Although  unit-cost  information  has  been  misused  and  abused  his¬ 
torically,  this  need  not  keep  unit-cost  analysis  data  from  providing 
valuable  information.  Proper  design  and  implementation  of  unit-cost 
analyses  and  a  thorough  understanding  of  the  nature  of  unit  costs  by 
decision  makers  can  result  in  better  decisions. 

4.  Cardinal-Weighting  Analysis 

Cardinal-weighting  analysis  departs  from  the  quantitative  approach 
except  for  the  calculation  of  direct  costs  for  each  alternative.  It 
relies  on  identification  and  specification  of  the  advantages  and  dis¬ 
advantages  of  each  alternative.  The  more  sophisticated  analyst  may 
consider  cardinal  weighting  as  amateurish,  but  listing  and  weighting 
advantages  and  disadvantages  of  alternatives,  even  if  done  on  a  value 
judgment  basis  alone,  will  most  probably  result  in  more  clarity,  honesty, 
and  inter-judge  reliability  in  alternative  selection. 

The  method  is  quite  simple,  but  can  be  very  effective  if  properly 
applied.  The  analyst  begins  by  listing  the  advantages  of  an  alterna¬ 
tive.  Every  advantage  for  an  alternative  is  weighted  by  awarding  each 
a  number  from  1  (lowest)  to  10  (highest)  in  value.  This  process  is 
followed  for  each  advantage  and  the  points  are  then  added.  The  procedure 
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is  repeated  by  listing  disadvantages  of  the  alternative  but  with  the 
values  of  the  weighting  scale  running  from  1  (least  disadvantageous) 
to  10  (most  disadvantageous).  A  cardinal-weighting  index  for  the  alter¬ 
native  can  now  be  obtained  by  subtracting  the  total  disadvantage  points 
from  advantage  points.  The  procedure  is  repeated  for  each  identified 
alternative  and  a  comparison  can  then  be  made  between  the  cardinal-weighted 
indices.  The  alternative  with  the  greatest  number  of  points  should  be 
the  most  valuable  one.  If  the  cost  of  that  alternative  is  the  same  or 
less  than  lower-weighted  alternatives,  the  recommended  decision  becomes 
obvious. 

This  procedure  is  useful  only  if:  (a)  the  analyst  lists  as  many 
advantages  and  disadvantages  as  possible  for  each  alternative,  (b)  the 
weights  are  fairly  assigned  to  each  advantage  and  disadvantage,  and  (c) 
the  decision  maker  realizes  that  the  results  of  the  analysis  are  only 
as  useful  and  valid  as  the  lists  are  comprehensive  and  the  weightings 
obj ective. 

This  procedure  will  at  least  clarify  the  issues  involved  when 
considering  alternative  approaches  or  programs  and  will  lend  assistance 
when  the  data  and  personnel  are  not  adequate  for  more  sophisticated 
analyses. 

5.  Ordinal  Ranking 

Ordinal  ranking  of  alternatives  is  an  analytical  procedure  in 
which  alternative  programs  are  ranked  in  order  on  the  basis  of  predeter¬ 
mined  criteria.  Criteria  can  be  determined  in  many  ways  by  many  and 
diverse  interests.  How  they  are  determined  is  as  important  as  the  ranking 
process.  They  might  be  developed  on  the  basis  of  total  or  direct  costs, 
time,  number  of  key  personnel,  or  learner  satisfaction.  Agreement  on 
each  criterion  measure  or  basis  of  the  ordinal  ranking  should  precede 
the  application  of  the  analysis.  Establishing  criteria  after  the  initial 
stages  of  the  analysis  have  been  completed  is  a  less  than  objective 
approach.  However,  if  new  and  valid  criteria  are  discovered  while  con¬ 
ducting  an  analysis,  there  is  nothing  to  prevent  additional  analyses 
being  made  using  the  new  criteria. 

6.  Best-Guess  Decision  Making 

Best-guess  decision  making  need  not  be  a  totally  unobjective 
process.  A  statement  of  the  reason  or  reasons  for  recommending  a  given 
alternative  can  be  based  on  considerable  research  and  evidence.  The 
reason  may  be  logical,  psychological,  political,  or  otherwise.  Much 
more  sophisticated  analyses  may  have  preceded  the  solution,  but  if  time 
and  other  factors  prevent  a  more  detailed  documentation,  a  simple  state¬ 
ment  of  why  a  particular  recommendation  was  made  will  be  of  some  help. 

Figure  2  is  intended  to  display  the  relationships  of  these  pre¬ 
liminary  analysis  concerns.  Although  the  arrangement  of  boxes  and  arrows 


indicates  a  rather  linear  sequence,  in  practice,  all  areas,  except  perhaps 
type  of  analysis  (Techniques),  must  be  continually  reviewed  and  considered. 

The  purpose  of  this  section  on  preliminary  concerns  is  to  empha¬ 
size  the  importance  of  careful  preplanning  and  prespecification.  To 
complete  the  picture,  a  look  at  the  way  various  components  are  combined 
to  form  subsystems  is  necessary.  A  complex  problem  may  require  that 
many  specific  analytical  models  be  developed.  Other  problems  may  only 
require  one  comprehensive  model.  Three  primary  analysis  subsystems  are 
discussed  in  the  following  section.  Each  is  an  example  of  an  important 
class  of  subsystems. 


IV.  COMPONENT  GUIDELINES 


The  analyst  must  report  various  kinds  of  data,  including  feasibility, 
context,  process,  effectiveness,  and  cost  data.  This  section  treats 
the  collection  and  organization  of  these  data  as  subsystems  of  the  total 
systems  analysis.  The  analyst  must  have  data  collection  methods  that 
provide  for  arraying  and  presenting  these  data  in  a  meaningful  way. 

The  data,  in  other  words,  must  become  information  for  decision  makers. 
Feasibility  analysis,  the  cost  model,  the  effectiveness  model,  and  other 
components  provide  the  framework  for  this  process. 

This  section  reviews  the  process  and  the  considerations  of  each  sub¬ 
system.  Feasibility  analysis  leads  to  a  tentative  determination  of  the 
level,  function,  purpose,  approach,  and  type  of  analysis.  It  defines 
the  parameters  of  the  system  to  be  used .  The  cost  model  delineates  and 
represents  the  dependence  of  the  cost  considerations,  cost  categorization, 
and  cost  estimating  relationships.  Instructional  process  and  context 
description  guidelines  are  provided  as  examples  of  terms  or  categories 
that  can  be  used  when  analyzing  those  vital  components.  The  effectiveness 
model  defines  its  elements  and  describes  methods  for  identifying  and 
predicting  the  measures  of  those  elements.  A  list  of  potential  cost 
and  effectiveness  criteria  is  included  at  the  end  of  this  section. 

A.  Feasibility  Analysis 

The  first  section  of  this  article  reviewed  systems  analysis.  Some 
mention  was  made  about  decisions  involved  in  designing  the  structure 
of  an  analysis  and  selection  of  an  approach,  but  little  was  said  about 
the  basis  for  those  decisions.  Why  is  one  approach  more  appropriate? 

Why  is  one  type  of  analysis  most  suitable?  What  needs  to  be  considered 
before  goals  and  objectives  are  specified?  What  alternatives  can  be 
considered?  Each  of  these  concerns  along  with  many  others  make  up  the 
domain  of  feasibility  analysis. 

Preliminary  feasibility  studies  can  lead  to  tentative  determination 
of  the  level,  function,  purpose,  approach,  and  type  of  analysis.  Once 
these  tentative  decisions  have  been  made,  emphasis  can  be  shifted  to 
the  specification  of  goals  and/or  objectives  for  each  alternative.  Based 
on  initial  analyses  and  generation  of  alternatives,  a  decision  maker 
often  is  forced  to  revise  these  program  goals  and  objectives  because 
of  unrealistic  expectations.  Often  both  objectives  and  alternatives 
will  need  to  be  revised  if  a  feasible  solution  is  to  be  identified. 

The  Importance  of  this  process  of  analysis,  generation  of  alternatives, 
and  subsequent  revision  cannot  be  overstressed.  Hitch  and  McKean  (1968) 
state:  "It  can,  in  fact,  be  argued  that  the  chief  gain  from  the  systema¬ 
tic  analysis  is  the  stimulus  that  it  provides  for  the  invention  of  better 
systems"  (p.  187). 

Feasibility  can  be  broken  into  two  general  categories  -  constraints 
and  capabilities.  Both  have  been  mentioned  but  additional  emphasis  is 
warranted. 
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1.  Constraints 


Constraints  can  be  defined  as  conditions  existing  both  within 
and  without  a  system  which  limit  the  achievement  of  specified  objectives. 
These  include  factors  associated  with  such  issues  as  time  pressures, 
manpower  availability,  political  exigencies,  and  other  resource  avail¬ 
ability.  Uncertainty  about  such  variables  as  future  fiscal  or  political 
stability  also  presents  serious  constraints  for  the  identification  of 
feasible  long-range  alternatives. 

Determination  and  selection  of  a  preferred  alternative  must  be 
made  from  a  set  of  feasible  alternatives.  Most  decision  makers  can  think 
of  highly  desirable  alternatives  that  for  obvious  reasons  are  not  possible. 
It  is  necessary  to  carefully  consider  any  constraints  or  factors  that 
separate  feasible  alternatives  from  those  that  are  not.  Judy  (1969) 
has  outlined  seven  general  constraint  categories  that  can  serve  as  review 
guidelines:  (1)  technological,  (2)  policy,  (3)  political,  (4)  organiza¬ 

tional,  constitutional,  and  legal,  (5)  resource,  (6)  target,  and  (7) 
imaginary. 

2.  Capabilities 

Just  as  constraints  have  the  potential  for  inhibiting  specified 
activities,  capabilities  have  the  capacity  for  assisting  the  accomplish¬ 
ment  of  identified  objectives.  Using  potential  inherited  facilities, 
equipment,  and  personnel  will  be  discussed  in  the  cost  model  section, 
but,  as  an  example,  consider  an  existing  instructional  system  as  a  feas¬ 
ible  future  alternative.  Rand  analysts  Carpenter  and  Haggart  comment 
on  such  a  possibility: 

In  educational  planning,  one  alternative,  simply  to  con¬ 
tinue  current  practice,  should  be  included  for  baseline 
data.  Although  this  alternative  will  usually  not  incur 
the  same  cost  as  will  the  innovative  programs  being  com¬ 
pared,  it  is  Important  to  know  its  projected  future  cost 
and  effectiveness  so  that  the  added  resource  and  cost  re¬ 
quirements  incurred  by  the  innovative  programs  may  be  es¬ 
timated.  Then  the  incremental  requirements  that  are 
associated  with  improved  effectiveness  will  be  known. 

(It  often  turns  out  that  these  incremental  requirements 
are  small  compared  with  the  requirements  simply  to  main¬ 
tain  current  practice,  even  though  they  might  seem  large 
when  considered  in  isolation.)  Thus,  the  first  step 
toward  cost-effectiveness  analysis  must  be  to  estimate 
the  future  resource  requirements  and  effectiveness  of 
current  programs  (pp.  12-13). 

Of  course,  this  applies  only  if  continuation  of  programs  is  to  be  considered 
or  if  current  programs  indeed  exist. 


3 .  Documentation 


Feasibility  assumptions  must  be  reported  as  part  of  an  analysis, 
in  the  same  manner  as  effectiveness  and  cost  data.  Regardless  of  the 
level,  function,  or  purpose,  upon  completion  of  a  thorough  analysis 
of  alternatives,  the  analyst  or  analysis  team  is  required  to  make  recommen¬ 
dations  based  on  available  data.  Recommendations  submitted  by  the  ana¬ 
lyst  (s)  must  be  accompanied  by  sufficient  documentation.  It  should  include 
assumptions  under  which  the  analysis  was  made,  purpose (s)  for  the  ana¬ 
lysis,  methodology,  approaches  used,  and  conclusions  reached.  Any  limi¬ 
tations  that  were  placed  on  the  analysis  or  that  should  be  placed  on 
comparisons  of  alternatives  should  be  listed  so  as  to  lessen  the  possi¬ 
bility  that  improper  assumptions  will  be  made  by  decision  makers. 

Carpenter  and  Haggart  (1969)  provide  some  guidelines  for  systems 
analysis  documentation: 

In  the  presentation  of  the  analysis,  the  qualitative 
considerations  should  be  identified.  This  includes 
both  those  taken  into  account  in  the  analysis  and  those 
that  could  not  be  made  an  integral  part  of  the  analysis 
itself.  It  is  important  to  present  the  results  of  the 
formative  quantitative  analysis  and  interpret  the  re¬ 
sults  with  special  attention  to  the  assumptions  and 
limitations  of  the  analysis.  In  addition,  the  analysts 
should  attempt  to  identify  the  important  qualitative  con¬ 
siderations  that  the  decision  makers  should  try  to  take 
into  account  (p.  5). 

B.  Cost  Model 

1.  Cost  Considerations 


As  with  the  general  systems  analysis  model,  design  of  the  cost 
model  requires  that  analysts  carefully  consider  many  factors.  Several 
of  these  (i.e.,  inheritance,  research  and  development,  and  time)  merit 
brief  discussion. 

a.  Inheritance.  Inheritance  consists  of  previously  procured 
tangible  assets  such  as  facilities,  equipment  (hardware),  and  software. 
The  cost  over  and  above  the  resources  that  can  be  inherited  is  the  addi¬ 
tional  or  incremental  cost  that  will  be  charged  to  a  new  system  or  pro¬ 
gram.  The  goal  of  the  analyst  is  to  minimize  this  incremental  cost  by 
building  on  existing  programs  where  possible  and  sharing  the  inheritance 
of  other  programs. 

b.  Research  and  Development .  Care  must  be  taken  to  distinguish 
between  research  and  development,  research  in  development,  and  research 
on  development.  Generally  speaking,  only  those  research  and  development 
costs  that  contribute  directly  to  program  goals  should  be  included  in 
the  cost  model.  If  resources  are  required  for  more  basic  research  on 
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development,  those  costs  should  be  singled  out  from  normal  research  and 
development  costs  and  reported  separately. 

c.  Time.  There  are  several  time  functions  that  must  be  considered 
In  education  and  training  cost  analysis.  Four  of  these  are  uncertainty, 
discounting,  depreciation,  and  obsolescence.  Each  has  an  Impact  on  total 
cost  but  determining  the  degree  of  impact  is  quite  difficult.  Few  ex¬ 
amples  of  applications  of  these  cost  functions  exist  in  general  educa¬ 
tion  literature,  but  military  and  industrial  applications  do  provide 
guidelines.  Uncertainty  was  mentioned  earlier  under  feasibility  analysis 
and  will  not  be  discussed  here,  but  the  importance  of  its  consideration 
cannot  be  overemphasized. 

McKean  (1958)  has  written  one  of  the  better  discussions 
on  the  various  approaches  and  rationale  to  discounting.  Hitch  ,and  McKean 
(1960)  present  several  arguments  in  support  of  the  process  of  discounting 
future  cost.  They  comment  that  acquisition  and  use  of  a  monetary  unit 
in  the  present  will  cost  less  than  acquiring  and  using  it  in  the  future. 

The  practice  is  supported  by  the  rationale  that  resources  available  today 
are  worth  more  than  those  same  resources  available  a  year  from  now. 

In  addition  to  the  question  of  whether  to  discount  or  not 
is  the  controversy  over  the  appropriate  rate  to  use.  Various  agencies 
use  discounting  techniques  in  their  program  plans,  but  the  rates  they 
use  to  discount  vary  widely.  The  rationale  used  for  establishing  a  given 
rate  is  usually  quite  subjective  but  is  an  important  concern,  particularly 
for  alternatives  offering  payment  or  expenditure  schedules  that  differ 
considerably  in  terms  of  time  and/or  amount.  In  the  case  of  Navy  training 
research,  this  issue  has  been  resolved  by  a  Defense  Economics  Analysis 
Council  Directive  (DoDINST  7041.3)  which  establishes  a  10%  discount  rate 
and  requires  it  to  be  used  in  all  economic  analyses  of  proposed  Defense 
investments.  This  does  not  prevent  the  conducting  of  alternative  analyses 
using  a  different  discount  rate  to  compare  potential  impacts  of  such 
practices. 

Depreciation  and  obsolescence  in  a  training  setting  are  valid 
time-based  cost  functions.  Determination  of  depreciation  schedules  based 
on  use,  output,  and  obsolescence  approaches  true  depreciation  costs  more 
closely  than  the  present  straight-line  depreciation  schedules  now  being 
used  by  most  cost  analysts.  When  technological  innovation  is  accelerated 
or  remains  at  a  high  rate,  the  expected  life  span  of  some  equipment  is 
shortened  by  the  increased  rate  of  obsolescence.  Depreciation  is  usually 
considered  a  fixed  cost,  but  consideration  must  be  given  to  the  fact  that 
increased  use  affects  the  value  of  equipment  by  changing  the  expected 
life  span.  Straight-line  computations  of  depreciation  (for  hardware  and 
software)  are  precise  but  they  are  often  misleading  when  used  as  the  only 
basis  for  depreciation  schedules.  Analysts  planning  future  programs  often 
omit  these  considerations  by  assuming  that  there  will  be  no  replacement 
during  a  fixed  time  period  and  that  there  will  be  no  residual  value  of 
equipment  (or  other  assets)  remaining  after  a  program  is  terminated. 
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Many  analysts  use  amortization  accounting  instead  of  depre¬ 
ciation.  This  requires  making  annual  payments  into  a  sinking-fund  to 
replace  original  equipment  at  the  end  of  its  economic  life.  Interest 
earned  on  money  in  the  sinking-fund  aids  in  accumulation  of  replacement 
money.  However,  this  process  is  not  usually  feasible  in  federal  or  other 
tax  supported  settings. 

A  useful  distinction  can  also  be  made  between  the  notions 
of  cost  savings  versus  cost  avoidance.  To  actually  save  hard  dollars 
in  a  relatively  closed  system  such  as  the  military  requires  careful 
consideration  of  those  areas  where  an  apparent  dollar  "saved"  could 
actually  be  a  resource  reallocated  to  some  other  purpose  or  activity. 
Saving  recruit  training  time  does  not  cut  total  billet  costs  since  those 
will  continue  and  be  charged  to  some  activity.  This  is  sometimes  referred 
to  as  cost  transfer. 

Proper  planning  and  implementation  of  an  efficient  training 
program  could,  however,  lead  to  an  avoidance  of  expensive  training  costs 
while  increasing  training  effectiveness  and  overall  system  productivity. 
Such  hard  dollar  savings  in  investment  or  operating  funds  are  the  most 
difficult  to  obtain  since  they  are  tied  directly  to  staffing  quotas, 
facility  construction  or  renovation,  and  the  conventional  threat  to 
empire. 

2.  Cost  Categorization 

One  of  the  first  tasks  to  be  performed  in  conducting  a  cost 
analysis  or  in  designing  a  cost  model  is  that  of  defining  and  listing 
all  the  major  cost  categories  of  resources  required  by  an  instructional 
alternative.  These  categories  serve  to  specify  the  segments  that  require 
analysis.  Identification  and  tabulation  of  such  interrelated  and  non¬ 
exclusive  cost  categories  as  joint  costs,  fixed  and  variable  costs, 
and  recurring  and  nonrecurring  costs  will  help  distinguish  between  in¬ 
structional  alternatives  on  a  functional  cost  basis.  In  this  way,  pre¬ 
dictive  costs  can  be  estimated  more  accurately  and  differential  cost 
effects,  both  in  magnitude  and  kind,  can  be  identified  as  program  elements 
within  and  between  alternatives. 

Results  of  any  cost  analysis  can  be  easily  biased  by  the  exclusion 
of  categories  or  the  inclusion  of  inappropriate  cost  categories.  Cer¬ 
tain  types  of  costs  (sunk,  discounted,  and  incremental)  can  easily  in¬ 
validate  an  otherwise  good  cost  model  if  handled  incorrectly. 

Sunk  costs  in  particular  are  difficult  to  handle  conceptually 
and  politically.  For  instance,  when  comparing  feasible  alternatives, 
one  option  may  well  be  to  continue  an  existing  system  that  has  already 
been  fully  implemented.  In  this  case,  an  equitable  analysis  would  only 
consider  the  future  cost  of  operating  and  maintaining  that  system,  not 
the  sunk  costs  that  were  allocated  in  the  past. 
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If  an  existing  system  or  subsystem  can  be  used,  its  inherited 
assets  from  prior  investment  could  give  that  option  an  initial  financial 
and  time-to- implementation  advantage.  Of  course,  the  comprehensive  life 
cycle  cost  implications  could  ultimately  paint  quite  a  different  picture. 

A  problem  arises  when  sunk  costs  are  not  documented  since  cost 
predictions  for  the  future  are  usually  based  upon  cost  profiles  from 
prior  and  ongoing  programs  or  activities.  Without  those  historical  cost 
records,  analysts  must  rely  on  other  potentially  less  valid  data  to 
construct  their  predictive  cost  models. 

In  order  to  accurately  predict  total  system  costs,  expenditures 
must  be  categorized  in  two  classes—nonrecurring  and  recurring — so  that 
long-  and  short-range  cost  predictions  can  be  made.  Nonrecurring  costs 
include  such  areas  as  research  and  development  activities  and  the  initial 
investment  to  implement  the  results  of  those  activities.  Among  the  list 
of  research  and  development  costs  are  preliminary  research  and  design 
studies,  design  and  development  of  subsystems,  and  system  evaluation 
costs.  Initial  investment  includes  costs  for  preliminary  training, 
equipment,  facilities,  and  installation  expenditures.  Recurring  costs 
are  composed  of  annual  operation  costs  needed  to  maintain  and  evaluate 
a  program.  This  includes  items  such  as  salaries,  maintenance  of  equipment 
and  facilities,  and  expendable  supplies. 

Analysts  also  must  determine  which  costs  included  in  an  alterna¬ 
tive  are  fixed  (i.e.,  those  that  would  not  change  because  of  changes  in 
level  of  activity)  and  which  are  variable  (i.e.,  those  that  respond  to 
alterations  in  quantity  or  quality  of  operation).  Such  a  separation 
provides  the  basis  for  estimations  of  how  costs  would  be  modified  or 
what  the  marginal  cost  effect  would  be  as  a  result  of  changes  in  enroll¬ 
ment,  instructional  strategy,  criterion  levels,  etc. 

A  method  of  cost  accounting  which  separates  total  system  costs 
into  three  general  classes  is  used  by  many  analysts.  The  three  classes 
can  represent  overlapping  time-phased  resource  requirements  of  an  instruc¬ 
tional  program.  The  classes  categorize  costs  by:  (a)  research  and  devel¬ 
opment — which  includes  all  costs  necessary  to  bring  the  system  into  readi¬ 
ness  for  operation  (nonrecurring),  (b)  investment — which  includes  all 
nonrecurring  costs  required  in  the  process  of  phasing  a  system  into  opera¬ 
tion  (e.g.,  facility,  equipment,  and  training  costs),  and  (c)  operation — 
which  includes  all  recurring  costs  necessary  for  the  operation  of  the 
system  (e.g.,  salary  and  allowances,  training,  and  maintenance  costs). 
Occasionally  a  subset  of  the  operating  cost  category  called  replacement 
is  Identified  in  order  to  separate  these  costs  from  normal  operating  costs. 

The  general  scope  of  a  cost  model  can  be  displayed  in  a  systematic 
manner  by  using  a  cost  table  or  matrix.  The  matrix  approach  classifies 
all  general  cost  elements  horizontally  and  system  or  phase  elements  ver¬ 
tically.  For  design  and  development  purposes,  the  matrix  could  include 
the  terminology  of  specific  hardware  and  software  components  as  well 


I 


28 


as  consideration  of  various  time  or  function  phases.  Examples  of  this 
matrix  format  and  a  list  of  activities  that  might  fall  within  each  cate¬ 
gory  are  provided  in  the  following  section. 


3.  Cost  Estimating  Relationships 

The  theory  of  cost  modeling  rests  basically  on  two  components 
or  relationships.  One  concerns  the  matter  of  deciding  what  is  to  be 
costed  and  charged  to  an  alternative.  This  task  consists  of  listing 
and  defining  the  appropriate  categories  of  total  dollar  cost.  The  other 
concerns  the  technique  of  cost  prediction.  These  relationships  are 
generally  termed  cost  estimated  relationships.  One  relationship  is  the 
estimate  of  the  cost  per  unit  of  each  resource.  The  second  type  is  the 
estimate  of  the  number  of  units  required  to  meet  the  necessary  prerequi¬ 
sites  for  each  alternative.  These  relationships  are  used  to  relate 
required  resources  to  their  costs.  In  developing  the  previously  discussed 
predictive  cost  model,  data  from  past  experience  or  ex  post  facto  cost 
analysis  information  can  be  used  to  derive  many  of  these  estimating 
relationships. 

Ex  post  facto  cost  analysis  or  cost  accounting  does  not  require 
cost  estimating  relationships  because  actual  costs  are  used  rather  than 
predicted  costs.  Data  from  cost  accounting  of  after-the-fact  cost  ana¬ 
lyses  do  provide  valuable  insight  and  guidelines  for  establishing  estima¬ 
ting  relationships  for  predictive  cost  models.  These  data  are  of  little 
use,  however,  when  a  newly  designed  alternative  includes  few  of  the 
components  of  the  operational  program.  Once  a  relationship  for  estimating 
the  unit  cost  of  a  resource  has  been  established,  either  from  prior  cost 
information  or  from  estimates,  the  next  step  is  to  attempt  to  determine 
how  many  units  of  this  resource  will  be  needed  to  meet  the  requirements 
of  the  alternative. 

A  predictive  cost  model  must  serve  as  a  device  for  estimating 
the  cost s  of  future  alternatives  by  translating  the  descriptions  of  these 
alternatives  into  requirements  for  resources  and  then  transforming  those 
requirements  for  resources  into  costs.  In  order  for  the  predictive  cost 
model  to  be  useful,  this  translation  of  alternatives  to  resources  to 
costs  must  be  relatively  simple  and  explicit  or  so  mechanized  and  flexible 
that  users  will  not  view  the  model  as  being  too  restrictive  or  difficult. 

A.  Limitations  of  Traditional  Accounting  Approach 

There  are  many  limitations  in  using  a  conventional  line-item 
budget  as  a  base  for  a  cost  model.  Briefly,  the  primary  problems  involved 
are:  (a)  the  difficulty  of  relating  the  budget  to  goals  or  objectives, 

(b)  the  difficulty  of  allocating  resources,  (c)  the  difficulty  of  pro¬ 
jecting  budgets  and  costs  beyond  the  current  budgeted  year,  (d)  the 
impossibility  of  relating  costs  to  outcomes,  and  (e)  the  lack  of  inte¬ 
gration  between  planning  and  control  (including  budgeting)  and  the  in¬ 
structional  or  program  element  of  an  instructional  system. 


Most  accounting  systems  are  designed  to  ensure  budgetary  com¬ 
pliance  rather  than  to  determine  program  costs  or  to  measure  instruc¬ 
tional  effectiveness.  Budgets  are  purposely  designed  to  identify  appro¬ 
priate  sources  of  funds  and  accounting  for  the  source  of  budgetary  expen¬ 
ditures  rather  than  to  assist  in  allocating  resources.  In  order  to  gen¬ 
erate  alternative  means  or  ways  to  accomplish  some  given  outcome,  a  de¬ 
cision  maker  must  be  able  to  relate  costs  or  expenditures  to  outcomes. 

When  this  cannot  be  accomplished,  the  generation  of  feasible  alternatives 
to  the  achievement  of  objectives  is  not  possible. 

Construction  and  use  of  the  traditional  budget  has  historically 
been  the  domain  of  budget  officers  or  finance  experts  and  communication 
between  them  and  those  responsible  for  instruction  has  been  minimal. 
Finance  experts  have  not  been  expected  to  be  responsible  for  instructional 
decisions  and  those  with  instructional  responsibilities  have  avoided 
planning,  allocating,  or  relating  costs  to  outcomes. 

The  purpose  of  the  Program  Planning  and  Budgeting  System  (PPBS) 
approach  is  to  provide  the  basic  and  essential  information  needed  by 
all  decision  makers  so  that  they  can  analyze  feasible  alternatives  and 
make  decisions  based  on  more  appropriate  data.  It  should  be  noted, 
however,  that  most  attempts  to  implement  such  a  system  in  education  and 
training  have  not  centered  around  the  information  needs  of  instructors 
or  program  designers. 

The  absence  of  a  PPBS-like  system  does  not  eliminate  the  possi¬ 
bility  of  an  analysis  study  being  conducted.  The  task  of  defining, 
costing,  and  analyzing  the  output  of  ongoing  alternatives  which  have 
not  been  defined  in  a  program  format  is  difficult,  however.  A  well 
designed  cost  analysis  model  will  assist  in  accounting  for  costs  of 
ongoing  programs  as  well  as  in  predicting  costs  of  potential  programs. 

5.  Cost  Analysis  References 

A  prospective  cost  analyst  seeking  assistance  from  the  literature 
can  easily  become  overwhelmed  by  the  volumes  devoted  to  costing,  cost 
accounting,  budgeting,  etc.  Although  not  specifically  written  for  educa¬ 
tion  and  training,  two  of  the  more  readable  and  useful  sources  are  the 
previously  mentioned  books  by  Fisher  (1971)  and  Seiler  (1969).  Appli¬ 
cation  and  adaptation  of  their  guidelines  and  procedures  should  assist 
greatly  in  designing  a  viable  cost  analysis  model. 

Fisher  relies  on  the  military  for  examples  of  applications  in 
his  book,  "Cost  Consideration  in  Systems  Analysis."  The  major  emphasis 
is  on  assessing  the  cost  implications  of  alternative  future  military 
capabilities.  He  does  not  discuss  cost  accounting,  cost  estimating  for 
future  operations,  or  cost  analysis  of  detailed  design  activities,  but 
much  of  what  is  included  is  applicable  to  these  areas.  Seiler's  rela¬ 
tively  brief  (108  pages)  text,  "Introduction  to  Systems  Cost-Effective¬ 
ness,"  assumes  a  basic  understanding  of  calculus  and  probability  theory, 
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but  mathematical  applications  are  minimal  and  literary  explanations  are 
well  done  and  ample.  From  the  view  point  of  a  cost  model  builder,  the 
most  important  guideline  to  be  gleaned  from  these  texts  is  that  the 
cost-related  information  for  each  alternative  must  be  transmitted  to 
decision  makers  in  a  format  that  will  accurately  represent  the  alternative 
and  permit  meaningful  comparisons. 

An  exceptionally  well  prepared  document  that  places  the  cost 
component  in  a  total  systems  analysis  perspective  is  a  DoD  publication 
titled  "Economic  Analysis  Handbook"  (undated).  This  introductory  manual, 
written  and  subsequently  revised  by  the  Defense  Economic  Analysis  Council, 
is  well  organized,  assumes  little  prior  knowledge,  and  complements  the 
contents  of  this  document. 

C.  Instructional  Process  Description 

In  order  to  design  or  evaluate  a  well  organized  instructional  system 
that  meets  the  goals  specified  by  the  training  objectives,  alternative 
methods  and  media  must  be  carefully  considered.  Adequate  specification 
of  these  processes  (sometimes  referred  to  as  pedagogies,  instructional 
strategies,  or  management  techniques)  and  their  interactions  is  essential 
to  properly  interpret  any  training  system  effectiveness.  If  these  com¬ 
ponents  can  be  identified,  then  instructional  process  comparisons  of 
alternatives  will  add  considerably  to  a  comprehensive  cost-effectiveness 
analysis. 

Several  means  of  selecting  these  components  are  covered  in  an  exten¬ 
sive  review  by  Spangenberg,  Reback,  and  Moon  (1973).  The  most  comprehen¬ 
sive  and  potentially  useful  set  of  procedures  developed  to  date  is  reported 
in  a  series  of  reports  published  by  the  Training  Analysis  and  Evaluation 
Group  (TAEG) .  One  of  this  series — "A  Technique  for  Choosing  Cost-Effective 
Instructional  Delivery  Systems"  (Braby  et  al.,  1975) — identifies  procedures 
and  models  for  choosing  between  media  alternatives. 

The  following  taxonomy  of  training  methods  should  aid  the  analyst 
in  adequately  specifying  or  identifying  employed  or  prospective  process 
strategies. 

1.  Passive  Presentations 


a.  Verbal  Delivery.  The  presentation  of  any  training  material, 
typically  by  an  individual  speaker  in  a  lecture  format  who  has  limited 
knowledge  of  student  comprehension  at  the  time  of  delivery.  That  is, 
the  student  feedback  rate  to  the  instructor  is  low  and  infrequent.  Film, 
closed-circuit  TV,  or  tape  recordings  are  alternative  presentation  methods. 
Although  support  is  not  essential,  visuals  and  other  minor  aids  such 

as  chalkboards  are  desirable. 

b.  Written  Delivery.  The  presentation  of  any  training  material 
by  print.  The  printed  word  has  the  advantage  of  being  readily  available 
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and  reusable.  The  organization  Is  fixed  and  may  not  always  be  suitable 
The  student  must  participate  to  obtain  the  necessary  information,  but 
no  overt  response  Is  required. 


c.  Demonstration.  An  exposition  designed  to  show  the  operation 
of  equipment  or  sequence  of  events  necessary  to  derive  a  solution.  This 
could  be  considered  a  subcategory  under  either  verbal  or  written  delivery 
where  a  specific  procedure  is  being  demonstrated.  This  form  of  delivery 
usually  involves  the  use  of  equipment  or  mock  ups.  Media  may  be  employed 
to  provide  the  demonstration. 

d.  Dramatization.  The  presentation  of  any  training  material 
which  has  been  dramatized  to  illustrate  some  objective  and  hold  the 
student's  attention.  This  can  require  extensive  preparation,  if  live, 
or  considerable  expense,  if  prepackaged  in  some  media  form  such  as  film 
or  closed-circuit  TV. 

2.  Student/Instructor  Interactive  Presentations 


a.  Discussion.  The  sharing  of  ideas,  relations,  experiences, 
and  other  types  of  information  by  a  group  of  students.  There  may  or 
may  not  be  extensive  questioning  by  a  group  leader  or  an  instructor,  or 
among  the  students.  The  group  leader  must  exercise  skill  in  directing 
the  discussion  for  meaningful  results. 

b.  Instructor  Query/ Student  Response.  An  interactive  presen¬ 
tation  with  multifold  purposes:  (1)  to  provide  the  instructor  with  feed¬ 
back  on  prior  training  efforts,  (2)  to  aid  the  student  in  understanding 
some  concept,  rule,  or  relationship,  and  (3)  to  allow  students  to  exchange 
ideas  on  a  more  structured  basis  than  discussion. 

c.  Student  Query/Instructor  Response.  A  means  of  allowing  the 
student  to  clarify  instructor  presentation.  This  type  of  interaction 
may  allow  the  instructor  to  partially  determine  the  adequacy  of  his  pre¬ 
sentation.  However,  much  time  can  be  spent  on  peripheral  questioning 
and  it  may  be  difficult  to  adequately  assess  student  understanding,  both 
qualitatively  and  quantitatively. 

3.  Print  and  Graphic  Methods 

a.  Programmed  Instruction.  Typically,  a  written  presentation 
requiring  frequent  student  responses  for  progress.  This  method  may  pro¬ 
vide  a  fixed  sequence  for  the  student,  or  allow  some  branching.  Although 
potentially  time  consuming,  100%  mastery  can  be  attained  by  repetitive 
cueing. 


b.  Computer-assisted  Instruction  (CAI) .  This  presentation  method 
can  include  elements  of  all  the  schemes  outlined  above.  The  reader  should 
be  aware  that  almost  every  method  has  been  computerized  to  some  degree 


and  more  careful  specification  of  methods  other  than  "CA1"  Is  probably 
necessary  for  adequate  process  description.  See  Fletcher  (1975)  for 
current  military  applications  of  CAI. 

4.  Func  t Iona 1  App ilea  t ions / Inter  actions 

a.  Laboratories.  An  interaction  between  student  and  some  physical 
equipment  under  specific  guidelines.  The  laboratory  may  be  set  up  in 

such  a  way  that  the  student  is  required  to  learn  to  operate  equipment 
or  to  use  equipment  in  learning  a  skill  such  as  foreign  languages  or 
chemical  analysis. 

b.  Simulation.  An  instructional  situation  that  provides  the 
student  with  practice  in  some  sample  of  the  actual  work  he  is  being 
trained  to  perform.  This  can  range  in  form  from  a  simple  wooden  mock  up 
to  a  complex  computer-driven  multiposition  situation  trainer. 

5.  Course  Management 

After  the  instructional  methodology  has  been  determined,  the 
strategy  used  in  applying  these  techniques  must  be  specified.  Instruc¬ 
tional  strategy  may  be  defined  as  a  system  for  sequencing  training 
methods  that  specifies  the  relationship  between  the  methods,  content, 
equipment,  and  facilities  necessary  to  meet  course  objectives.  Some 
of  these  relations  may  be  under  student  control.  Careful  specification 
of  the  training  strategy  is  essential  in  cost-effectiveness  analysis, 
since,  without  it,  little  can  be  said  about  the  relationship (s)  between 
inputs  (context,  resources,  etc.)  and  the  outcomes  of  analyzed  alter¬ 
natives. 


The  following  variables  must  also  be  specified: 

a.  Student  Throughput.  Unless  throughput  is  sufficiently  high, 
even  the  most  elegant  strategy  may  be  judged  inefficient  because  of  high 
unit  costs. 


b.  Appropriateness  of  Instructional  Methods.  The  method  must 
exploit  both  the  technology  and  the  skill  to  be  learned.  For  example, 
using  a  high  fidelity  simulator  to  train  a  part-task  operator  may  not 
be  cost  justifiable. 

c.  Student  Management.  Included  in  this  category  are  several 
variables  that  can  strongly  effect  learner  outcomes.  Student  motivation 
would  fall  under  this  heading,  as  would  class  heterogenity,  outside  ac¬ 
tivities  that  minimize  study  time,  and  monitoring  student  progress. 

This  last  variable  can  be  particularly  important  if  the  alternative  is 
individualized . 
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D.  Instructional  Context  Description 

Whether  the  intent  is  to  design  a  new  instructional  program,  to  evalu¬ 
ate  an  ongoing  program,  or  to  compare  several  existing  or  potential  instruc¬ 
tional  alternatives,  it  is  expected  that  an  individual  or  team  can  obtain 
descriptive  data  that  identifies  the  environment  within  which  these  alter¬ 
natives  exist.  Such  data  would: 

1.  Provide  a  baseline  of  status  information  that  describes  the  domain 
of  concern,  e.g.,  institutional  setting,  learner  and  staff  characteris¬ 
tics,  etc. 

2.  Identify  the  needs,  goals,  mission  statements,  objectives,  prob¬ 
lems,  and/or  opportunities  that  apply  to  the  instructional  system  under 
examination. 

3.  Provide  a  basis  for  stating  change  objectives,  evaluative  criteria, 
and  standards  for  judging  the  relative,  absolute,  or  comparative  merits 

of  the  instructional  system(s). 

In  order  to  design,  evaluate,  and/or  compare  instructional  alterna¬ 
tives,  it  is  usually  desirable  (but  not  always  possible)  for  them  to 
be  located  in  a  common  site,  to  contain  similar  content  goals,  to  include 
students  with  comparable  entering  characteristics,  etc.  Examples  of 
Navy  context  descriptors  include  school  class  (A,  C,  E,  etc.),  course 
classification  definitions,  and  course  content  outlines.  Without  a  clear 
program  or  course  description,  criteria  may  be  selected  which  will 
lead  to  invalid  comparisons  or  evaluations.  The  more  precise  the  descrip¬ 
tion  of  instructor  expectations,  learner  outcomes,  and  program  components, 
the  greater  the  likelihood  of  valid  comparisons. 


A  brief  example  of  a  basic  course  outline,  including  a  subset  of 
learner  tasks  follows.  Explictly  specified  learner  outcomes  are  a  much 
more  useful  set  of  descriptors,  but  these  may  not  always  be  available. 


SONAR  TECHNICAL  INTERMEDIATE  ELECTRONICS  COURSE 
General  Content  Outline 

I.  Basic  Electrical  Theory 

A.  Mathematics 

B.  Geometry  and  Trigonometry 

C.  Safety 

D.  Basic  Electrical  Theory  and  DC  Fundamentals 

E.  Basic  AC  Theory 

F.  AC  Circuits 

G.  AC  Generators  and  Motors 

II.  Acoustical  Energy 

I.  Equipment  Performance 

II.  Basic  Tube  Type  Electronic  Devices 

A.  Electronic  Tube  Theory  and  Operation 

B.  Power  Supplies 

C.  Amplifiers 

D.  Oscillators 

E.  General  Electronic  Devices 

F.  Troubleshooting  and  Equipment  Performance 

III.  Basic  Transistor  Theory  and  Equipment 

A.  Basic  Transistor  Theory  and  Operation 

B.  Amplifiers 

C.  Oscillator  Circuits 
(trainee  outcomes: examples) 

1.  Describe  the  operation  of  a  transistorized  Armstrong  Oscilla¬ 
tor. 

2.  Define  the  function  of  a  transistorized  Hartley  (series  or 
shunt  fed)  oscillator. 

3.  List  the  different  bias  requirements  for  a  transistor  oscilla¬ 
tor  compared  to  a  tube-type  oscillator. 

4.  Describe  the  purpose (s)  of  a  transistorized  a  -,  mono  -, 
and  bi-stable  multivibrator. 

D.  General 

E.  Simple  Logic  Circuits 

F.  Servicing  Equipment  and  Equipment  Performance 

IV.  Electromechanical  Devices 

A.  Basic  Synchro  and  Servo  Systems 

B.  Servo  Motors 

C.  Miscellaneous  Electromagnetic  Devices 

D.  Computer  Circuits 

V.  Introductory  Computer  Theory 

A.  Basic  Computer  Functions 

B.  Basic  Fire  Control  Symbols 

C.  Analog  Computer  Mathematics  and  Mechanics 


E .  E ffectiveneBs  Model 


The  analysis  of  effectiveness  of  training  and  education  programs  or 
alternatives  should  be  as  vigorously  developed  as  the  analysis  of  the 
resources  used  to  implement  them.  Uses  of  systematic  techniques  for 
planning  an  instructional  alternative  should  lead  to  the  expansion  and 
improvement  of  ways  to  relate  the  quantity  and  quality  of  the  output 
of  the  alternative  to  the  resources  used  to  create  that  output.  Decision 
makers  also  need  criteria  other  than  cost  for  measuring  quality  or  use¬ 
fulness.  The  criteria  for  these  outputs  (other  than  cost)  can  be  grouped 
under  the  general  term  effectiveness.  Just  as  cost  is  a  measure  of  the 
resources  that  comprise  an  alternative,  effectiveness  is  a  measure  of 
the  output  of  that  alternative.  If  possible,  effectiveness  is  measured 
in  terms  of  performance  or  change,  but  it  can  also  be  measured  in  general 
terms  with  qualitative  interpretations. 

Like  any  cost  model  or  procedure,  the  effectiveness  model  can  be 
described  in  terms  of  its  elements  and  methods  for  identifying  and  pre¬ 
dicting  the  measures  of  those  elements.  The  kinds  of  components  that 
can  be  used  are  as  numerous  as  the  kinds  of  systems  and  measures.  The 
basic  elements,  by  necessity,  must  oversimplify  a  very  complex  real  world. 
What  is  required  of  these  components  is  that  they  usefully  express  the 
relation  between  measures  and  performance,  or,  most  specifically,  the 
capability  of  the  alternative  to  achieve  specified  objectives. 

Each  instructional  alternative  may  be  comprised  of  multiple  objec¬ 
tives  aimed  at  achieving  specified  goals.  It  is  important  to  determine 
whether  those  objectives  are  unique  or  if  they  actually  form  a  hierarchy 
with  a  single,  assessable  objective  at  the  top.  If  a  single  measure 
of  effectiveness  can  be  determined  from  a  set  of  objectives,  it  is  much 
easier  to  interpret  and  report.  Quade  (1964)  suggests  several  methods 
for  combining  and  eliminating  multiple  objectives:  "...elimination  of 
any  objective  that  is  important  only  as  a  means  to  another  objective 
...selection  of  a  higher  level  objective  to  which  all  of  the  competing 
objectives  are  means  (pp.  159-160)." 

1.  Evaluative  Criteria 


In  order  to  comprehensively  evaluate  an  ongoing  instructional 
system  or  to  compare  feasible  alternatives,  standards  for  judgment  must 
be  established.  Such  standards  are  necessary  to  help  determine  the  rela¬ 
tive  importance  of  various  outcome  measures  and  to  assist  in  establishing 
appropriate  or  acceptable  levels  of  those  measures. 

An  instructional  system  evaluation  plan  must  consider  not  only 
learner  performance,  but  also  cost,  time,  and  value  criteria.  Effective¬ 
ness  criteria  stated  in  terras  of  learner  performance  must  reflect  stand¬ 
ards  or  norms  selected  by  decision  makers.  The  process  used  to  establish 
standards  and  validate  measures  oftentimes  requires  considerable  atten¬ 
tion  by  many  parties. 


36 


Consideration  of  costs,  as  stutod  previously,  must  be  made  in 
relation  to  other  factors  such  as  resources,  outcomes,  and  constraints. 
Time  is  also  closely  related  to  cost,  effectiveness,  and  efficiency. 
Consideration  of  time  in  relation  to  learners  includes  measures  of  total 
lapsed  time  (beginning  and  ending  dates),  learning  time  (actual  time 
spent  in  learning),  and  retention  over  time  (reliability-stability). 

Value  and  utility  are  the  ultimate  criteria  since  the  payoff  of  an  alter¬ 
native  must  be  eventually  measured  in  those  terms. 

The  value-added  concept  of  learner  assessment  is  also  an  important 
consideration.  The  concept  does  not  lead  to  judgment  of  alternatives 
on  their  outputs  alone,  but  by  their  outputs  relative  to  their  inputs. 
Learner  change  (in  relation  to  learner  capability)  attributable  to  the 
effects  of  an  alternative  is  the  sought-after  measure  of  educational 
value  added. 

As  stated  previously,  no  single  measure  of  effectiveness  can 
tell  the  entire  story  of  the  value  or  worth  of  an  alternative  because 
any  instructional  alternative  generally  promotes  several  different  kinds 
of  change  in  learners.  Since  the  changes  are  varied  and  differ  in  kind, 
no  single  proxy  measure  can  be  used  to  represent  all  the  changes  attri¬ 
butable  to  the  alternative.  The  effectiveness  model  should  therefore 
allow  output  measures  to  be  displayed  as  a  set  of  measures  and  indicators. 

2.  Prediction 


There  is  one  major  concern  about  the  design  and  use  of  effective¬ 
ness  modelling  procedures.  A  simulation  of  costs  and  effectiveness 
of  most  instructional  alternatives  is  difficult  since  valid  predictive 
relationships  between  instructional  inputs  and  process  and  instructional 
results  or  outcomes  are  nonexistent.  Ex  post  facto  effectiveness  analysis 
is  less  speculative  but  still  contains  problems  of  measurement,  attribution, 
and  comparison.  Techniques  for  effectiveness  modelling  are  relatively 
underdeveloped  both  in  military  and  civilian  contexts.  Considerable 
empirical  and  conceptual  research  is  still  required  before  it  will  be 
possible  to  identify  useful  predictive  relationships  between  program 
components  and  outcomes. 

Perhaps  the  best  that  can  be  accomplished  in  predictive  compari¬ 
sons  of  the  effectiveness  of  instructional  alternatives  are  qualitative 
estimates  of  the  degree  to  which  each  alternative  will  contribute  to 
the  achievement  of  specified  objectives.  Although  prior  experience  and 
data  from  related  studies  will  help  with  these  estimates,  they  will 
nevertheless  remain  crude  estimates  for  the  foreseeable  future. 

An  example  of  an  effectiveness-feasibility  matrix  reporting  format 
can  be  found  in  the  summary  report  by  Doughty  and  Stakenas  (1973) .  In 
it,  various  critiera  are  considered  across  four  feasible  instructional 
alternatives.  Quantitative  and  qualitative  data  are  thus  reported  in 
such  a  way  that  a  range  of  levels  of  decision  makers  can  easily  compare 
and  judge  for  themselves. 
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F.  Cost  and  Effectiveness  Evaluation  Criteria 


The  following  exemplary  outline  or  taxonomy  of  evaluative  criteria 
has  been  developed  to  show  how  a  wide  range  of  data  could  be  used  in 
an  outcome  or  effectiveness  analysis  in  a  total  C-E  analysis.  This  list 
is  not  meant  to  be  exhaustive,  but  rather  to  illustrate  the  categories 
of  measures  that  are  typically  encountered. 

TAXONOMY  OF  COST  AND  EFFECTIVENESS  EVALUATION  CRITERIA 

I.  Learner-Trainee  Outcomes 

A.  Cognitive  and  Psychomotor  Data 

1.  Prerequisite  (Entering  Behavior)  Test  Data 

2.  Pretest  Data 

3.  Criterion-Referenced  Posttest  Data 

4.  Norm-Referenced  Posttest  Data 

5.  Pre-Post  Test  Gain  Score  Data 

6.  Performance  Data 

7.  Criterion-Referenced  Retention  Data 

8.  Norm-Referenced  Retention  Data 

9.  Transfer  of  Training  Data 

B.  Affective  Data 

1.  Preinstruction/Training  Attitude 

2.  Preinstruction/Training  Satisfaction 

3.  Postinstruction/Training  Attitude 

4.  Postinstruction/Training  Satisfaction 

5.  Psychophysiological  measures 

C.  Other  Outcomes 

1.  Positive  Side  Benefits:  System  and  Learner 

2.  Negative  Side  Benefits:  System  and  Learner 

3.  Absenteeism 

II.  Eff iciency/Tirae  Criteria  and  Considerations 

A.  System 

1.  Front  End  Analysis  Time 

2.  Design/Developraent/Validation  Time 

B.  Learner 

1.  Chronological  -  Lapsed  Time 

2.  Total  Active  Instructional  Time:  Time  to  Learn 

3.  Retention  Time:  Intervening  time  between  test  and  retest 


C.  Other  Considerations 


1.  Time  Sequence  Constraints 

2.  Repetition/Replication  Spacing  and  Requirements 

3.  Replacement  Schedule 

4.  Depreciation:  Physical  Life  Cycle 

5.  Obsolescence:  Technological  Life  Cycle 

6.  Operational  Utility:  Content/Doctrine  Stability-Operating 
Lifetime 

Feasibility 

A.  Appropriateness-Relevance 

B.  Availability 

C.  Capability 

D.  Convenience 

E.  Cooperability 

F.  Cooptability 

G.  Criticality 

H.  Dependability:  Hardware 

I.  Evaluatibility-Measurability 

J.  Exportability 

1.  Diff usability 

2.  Marketability 

K.  Pervasiveness 

L.  Political-Legal  Constraints 

M.  Reliability:  Assessments 

N.  Social-Moral  Concerns 

O.  Uniqueness 

P.  Utility 

Q.  Validity:  Goals,  Assessments 
Dollar  Cost  Criteria 

A.  Research  and  Development  Costs 

B.  Investment  and  Production  Costs 

C.  Replacement  Costs 

D.  Operation  Costs:  One  Cycle  and  Annual 

E.  Operation  Costs:  Lifetime 

F.  Total  Lifetime  Dollar  Costs  (Life  Cycle  Costs) 


V.  COST  CATEGORIZATION  GUIDELINES 


A  generalizeable  cost  categorization  scheme  that  vill  meet  all  the 
needs  of  cost  analysts  has  not  been  and  likely  will  not  be  identified. 

The  number  of  categories  and  the  extent  of  detailed  cost  data  contributing 
to  each  category  should  be  determined  by  a  user's  judgment  of  the  func¬ 
tion  and  purpose  of  the  analysis.  Considerable  differences  could  be 
noted  between  analyzing  ex  post  facto  costs,  for  instance,  of  an  existing 
Navy  'A'  school  training  program  and  predicting  total  dollar  costs  for 
an  innovative  shipboard  oriented  instructional  strategy.  If  combinations 
of  functions  (control,  planning,  evaluation,  and  development)  are  to 
be  considered,  even  greater  effort  should  be  devoted  to  determining  what 
data  are  to  be  collected,  how  they  will  be  obtained,  and  who  will  have 
access  to  the  results.  This  last  concern  is  vital  because  that  decision 
often  influences  the  validity  of  obtained  and/or  reported  data.  The 
threat  factor  is  always  a  tremendous  influence  on  data  availability  and 
accuracy. 

Figure  3  illustrates  some  of  the  decision  points  in  a  "typical”  Navy 
training  cost  analysis.  The  term  initiator  was  selected  to  head  the 
first  column  since  may  analyses  are  conducted  for,  but  not  necessarily 
by,  higher  level  decision  makers.  Navy  trainers,  for  instance,  may  use 
the  results  of  cost  analyses  for  personnel  planning  and  course  development 
purposes,  but  the  same  results  or  aggregates  thereof  may  also  be  requested 
by  research  directors  or  DoD  planners. 

Program  focus  or  scope  will  depend  primarily  on  the  analyst's  (or 
initiator's)  perception  of  function  and  purpose.  For  example,  a  school 
curriculum  coordinator  may  be  concerned  with  designing  and  developing 
an  innovative  instructional  strategy.  That  strategy,  defined  as  a  pro¬ 
gram,  could  become  the  focus  of  a  total  analysis  designed  to  predict 
dollar  costs  and  trainer  time  as  well  as  trainee  time,  satisfaction, 
and  achievement.  These  data  could  be  used  by  that  coordinator  for  plan¬ 
ning  purposes  and  as  development  specifications. 

A  division  director,  on  the  other  hand,  may  define  a  program  encom¬ 
passing  all  schools  within  his  unit.  In  this  instance,  an  ex  post 
facto  description  approach  may  provide  data  useful  for  several  functions, 
but  careful  consideration  will  have  to  be  given  to  the  procedures  used 
to  collect  those  data.  An  even  more  important  decision  will  be  the 
announced  and  unannounced  use(s)  of  the  data  (i.e.,  performance  review, 
course  evaluation,  budget  allocation,  internal  planning)  by  that  director. 
Validity  becomes  much  more  of  an  issue  when  both  the  purpose  of  an  analysis 
and  use  of  resultant  data  are  perceived  as  threats  by  those  supplying 
needed  data. 

Before  any  cost  analysis  summary  can  be  constructed,  the  analyst 
must  aggregate  specific  costs  into  larger  cost  categories.  The  following 
scheme  is  designed  so  that  cost  data  may  be  aggregated  by  hand  calcula¬ 
tion.  However,  specific  cost  categories  are  open-ended,  and  a  complete 
listing  is  not  possible  because  of  the  uniqueness  of  particular  programs. 
The  following  outline  lists  activities  and  categories  that  have  been 
used  in  a  number  of  cost  studies. 
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DECISION  POINTS  FOR  COST  ANALYSIS 


COST  CATEGORIZATION  FORMAT 

Examples  of  Activities  and  Categories  for  Cost  Analysis 
Activities 

A.  Research  and  Development 

1.  Needs  assessment  -  front  end  analysis  -  initial  planning 

2.  Task  analysis  -  job  analysis 

3.  Curriculum  design 

4.  Prototype  development  and  testing 

5.  Formative  evaluation  -  preliminary  product  and  program  review 

6.  Materials  validation 

7.  Training  program  and  equipment  development 

8.  Initial  personnel  recruitment  and/or  training 

B.  Investment  and  Production 

1.  Acquisition  -  Installation  -  Start  up  costs 

2.  Procurement  of  initial  stock  of  training  hardware  and  software 

3.  Duplication  of  production  masters 

4.  Construction  -  Renovation  of  facilities 

5.  Purchase  of  initial  spare  components 

6.  Modifications  of  existing  systems 

7.  Initial  deployment  of  training  hardware  and  software 

8.  Initial  dissemination  of  duffusion  and  implementation  activities 

C.  Replacement 

1.  Attrition 

2.  Replacement  as  a  result  of: 

a.  Obsolescence 

b.  Depreciation:  Normal  use 

c.  Theft  -  Vandalism  -  Breakage 

3.  Periodic  (scheduled  or  unscheduled)  updating  of: 

a.  Content  -  materials 

b.  Equipment 

c.  Procedures  -  management 

D.  Operation 

1.  Personnel 

a.  Instructional:  salary  -  travel  -  benefits  (including 
retirement) 

b.  Administrative  -  managerial 

c.  Maintenance  -  support 

d.  Students:  salary  -  travel  -  benefits 
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2.  Materials  -  consumables 

3.  Ongoing  training  and  evaluation 

4.  Ongoing  distribution  -  deployment  of  hardware  and  software 

5.  Facilities 

6.  Overhead 

11.  Cost  Categories 


A.  Personnel:  Salaries  and  Benefits 

1 .  Instructional  staff 

2.  Support  staff:  Non  instruction  -  secretarial 

3.  Program  administrative  -  managerial  personnel  -  supervisors 

4.  Maintenance  staff 

B.  Hardware 

1.  Simulators  -  trainers 

2.  Audiovisual  equipment 

C.  Software 

1.  Instructional  materials  and  supplies 

2.  Training  aids 

3.  Expendable  materials 

4.  Training  manuals,  technical  manuals 

D.  Facilities 

1.  Classrooms 

2.  Laboratories 

3.  Self-instructional  facilities 

4.  Administrative  -  managerial  -  support  facilities 

E.  Institutional  Overhead/Administration 

1.  Agency  -  institutional  management 

2.  Libraries 

3.  Computer  facilities 

4.  Contracted  services  -  consultants 

5.  Institutional  overhead 
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A.  Cost  Activities 


Program  activities  are  a  major  focus  of  any  categorization  system. 
Several  referenced  models  employ  functional  category  descriptors  similar 
to  those  in  the  outline,  but  that  is  not  to  say  that  these  categories 
are  the  only  ones  that  might  be  acceptable.  Several  referenced  reports 
include  sections  containing  extensive  specification  and  definition  of 
program  activity  categories.  For  our  purposes,  a  brief  overview  should 
suffice. 


1.  Research  and  Development 

Costs  include  all  funds  expended  to  bring  an  ongoing  or  planned 
program  (alternative-strategy)  into  readiness  for  implementation.  R&D 
expenditure  for  front-end  analysis,  design,  development,  formative  evalua¬ 
tion,  staff  (preservice)  training,  and  procurement  of  R&D  materials  and 
equipment  are  essentially  one-time  nonrecurring  costs  and  should  be  so 
identified.  Some  nonrecurring  costs,  however,  should  not  be  assigned 
to  an  R&D  phase  or  function  but  to  an  investment  and  production  func¬ 
tion.  This  second  major  activity  includes  all  dollar  costs  required 
to  phase  a  program  into  operation.  These  include  costs  for  facility 
renovation  or  procurement,  instructional  equipment  acquisition,  and  pro¬ 
duction/duplication  of  instructional  materials.  If  materials  are  commer¬ 
cially  available,  they  are  also  charged  to  this  activity. 

2.  Facilities 


Many  schemes  include  some  prorated  estimate  of  facilities  or 
overhead  cost  in  either  the  investment  or  operation  phase.  Considerable 
study  still  needs  to  be  given  this  particular  area,  but  if  feasible  alter¬ 
natives  being  considered  all  require  similar  existing  facilities,  then 
these  costs  can  conceivably  be  classified  as  a  constant  and  perhaps  be 
excluded  from  the  analysis.  This  will  not  be  the  case  if  new  or  addi¬ 
tional  facilities  or  other  overhead  are  required.  Appendix  A  contains 
a  brief  discussion  of  the  range  of  alternatives  for  costing  instructional 
space. 


3.  Replacement 

Many  models  also  include  various  replacement  costs  for  equipment 
and  materials  in  the  operation  cost  category.  This  may  not  be  appropriate 
if  alternatives  being  compared  differ  drastically  in  this  area.  To 
include  the  replacement  cost  of  a  computer  or  a  simulator  in  the  general 
operating  category  may  be  highly  misleading.  Within  this  area,  predicting 
depreciation  and  obsolescence  rates  for  instructional  materials  and  equip¬ 
ment  is  often  described  as  an  art  form.  The  traditional  estimate  of 
a  5-to-10  year  lifespan  for  training  hardware,  for  instance,  is  based 
on  "normal  usage."  Factors  such  as  maintenance  schedules,  amount  of 
use,  type  of  use,  user  sophistication,  and  theft  rates  should  be  con¬ 
sidered  and  somehow  factored  into  any  replacement  estimate.  Obsolescence 
predictions  for  software  will  depend  upon  such  variables  as  content  sta¬ 
bility,  style  changes  in  visuals,  format  (hardbound,  cassette,  workbook), 
and  usage. 
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A  considerable  portion  of  the  recurring  costs  for  any  program 
or  alternative  should  be  included  in  the  operating  cost  category. 
Instructional  personnel,  periodic  maintenance,  expendable  supplies,  sum- 
mative  evaluation,  in-service  training  and  managerial  overhead  are  all 
recurring  costs  and  should  be  classified  as  operational  costs.  In  a 
conventional  setting,  one-cycle  operation  costs  might  be  the  funds  re¬ 
quired  to  maintain  one  program  (course)  for  one  complete  iteration  (cycle) . 
Innovative  programs  will  present  much  more  of  a  challenge.  One  cycle 
of  an  individualized,  non-time-based,  ship-based  program  is  not  so  easily 
segmented,  categorized,  and  costed.  Considerable  review,  testing,  and 
revision  will  be  necessary  before  any  existing  conventional  procedures 
will  be  useful. 

A  predictive  model  may  make  use  of  ex  post  facto  data  to  establish 
cost  estimating  relationships.  It  is  therefore  important  that  a  general 
cost  summary  be  suitable  for  both  descriptive  and  predictive  data.  When 
lifetime  operation  costs  are  required  (and  they  are  most  important,  albeit 
speculative)  conventional  programs  present  much  less  of  a  problem  than 
newly  evolving  innovative  types.  Although  predicting  the  obsolescence 
rate  of  course  content,  the  number  of  times  per  year  a  course  is  offered, 
and  the  number  of  years  it  may  continue  to  be  offered  (before  major 
revision)  is  a  challenge,  it  does  not  compare  to  the  difficulty  of  estimating 
the  scope,  duration,  and  number  of  cycles  in  the  "life"  of  a  modularized 
individualized  program. 

Estimating  life  cycle  costs  for  Navy  personnel  has  been  greatly 
simplified  by  analysts  in  the  Personnel  Plans  Division  of  the  Bureau 
of  Naval  Personnel.  A  sporadically  published  report,  "Navy  Military 
Manpower  Billet  Cost  Data  for  Life  Cycle  Planning  Purposes"  (which  was 
last  published  in  1973),  contains  comprehensive  annual  billet  costs  for 
both  enlisted  and  officer  personnel.  Included  in  the  reported  figures 
are  actual  and  estimated  costs  for  retirement  and  other  fringe  benefits. 
Needless  to  say,  these  particular  costs  can  be  a  significant  factor  when 
comparing  alternatives  with  differing  degrees  of  labor  intensiveness. 

In  addition.  Appendix  B  contains  a  discussion  of  alternatives  for  de¬ 
termining  personnel  costs  for  an  operational  instructional  system. 

B.  General  Guidelines 

A  reasonable  guideline  for  after-the-fact  or  predictive  cost  analysis 
is  to  devote  attention  to  any  particular  category  according  to  the  pro¬ 
portion  of  the  total  budget  reflected  by  that  component.  Obviously, 
personnel  costs  in  most  training  and  education  contexts  will  account 
for  a  large  percentage  of  any  budget  so  an  analyst's  energies  should 
reflect  that  fact.  Fortunately,  personnel  costs  hold  few  surprises  or 
computational  difficulties  and  may  usually  be  guided  by  past  cost  experi¬ 
ence  or  programs  that  employ  similar  types  of  personnel  configurations. 
However,  as  a  proposed  program  deviates  more  and  more  from  conventional 
practice,  the  utility  of  conventional  program  data  for  guiding  cost 
prediction  diminishes. 


A  category  that  represents  a  small  proportion  of  a  total  budget 
deserves  less  attention  since  even  relatively  large  errors  in  accounting 
for  or  estimating  these  costs  will  not  significantly  effect  the  total 
cost  figure.  The  temptation  to  diligently  obtain  the  latest  cost  figures 
for  paper  clips  and  pencils  and  settle  for  rough  estimates  of  expensive 
computer  time  should  always  be  resisted. 

The  outline  provided  contains  one  general  functional  categoriza¬ 
tion  scheme  for  Navy  education  and  training  R&D  cost  analysis.  It  is 
eclectic  in  that  it  includes  components  found  in  several  but  not  all 
costing  schemes.  Many  such  schemes  do  not  separate  the  replacement 
function  from  operation  activities,  but  that  decision  can  be  made  once 
data  are  obtained  and  levels  of  aggregation  can  be  considered. 

C.  Program  Cost  Analysis  Summary 

Once  cost  data  have  been  categorized,  collected,  and  processed,  one 
useful  way  to  array  or  report  the  results  is  to  construct  a  summary  matrix. 
Such  a  matrix  helps  transform  the  data  into  information  by  identifying 
recurring  and  nonrecurring  costs  as  well  as  fixed  and  variable  costs. 

It  also  helps  decision  makers  to  review  and  compare  instructional  alterna¬ 
tives  on  a  functional  cost  or  program-oriented  basis. 

In  Figure  4,  general  cost  categories  are  listed  on  the  horizontal 
axis.  These  are  obviously  gross  categories,  but  this  is  intended  to 
be  used  as  a  summary  or  a  display  of  aggregated  cost  data.  The  vertical 
axis  displays  the  previously  defined  activities  with  a  subtotal  now  added 
to  emphasize  and  isolate  nonrecurring  dollar  costs. 


The  preceding  sections  have  illustrated  the  all-important  concern 
for  initial  emphasis  on  the  function (s)  and  purpose (s)  of  any  cost-effec¬ 
tiveness  analysis.  Cost  categorization  and  matrix  reporting  schemes, 
such  as  the  one  shown  in  Figure  4,  are  important  tools  but  should  not 
be  employed  before  the  questions  or  problems  are  well  defined. 


CATEGORIES  PERSONNEL  HARDWARE  SOFTWARE  FACILITIES  INSTITUTIONAL  TOTAL 

OVERHEAD/ 

ADMIMISTRATION 


VI.  CONCLUSIONS 


The  use  of  cost-effectiveness  analysis  in  establishing  and  reviewing 
military  training  programs  is  becoming  increasingly  necessary  as  all 
Department  of  Defense  budget  allocations  come  under  closer  scrutiny. 

The  guidelines  and  examples  presented  in  this  report  are  designed 
specifically  for  the  Navy  training  and  training  R&D  communities  and 
provide  an  introduction  into  the  field  with  additional  sources  of  in¬ 
formation  clearly  identified  so  that  the  reader  interested  in  applying 
these  guidelines  should  have  an  adequate  basis  for  either  conducting 
in-house  cost-effectiveness  studies,  or  monitoring  contractual  studies. 
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The  following  nine  references  have  been  carefully  selected  from  a  com¬ 
prehensive  review  of  cost-effectiveness  analysis  related  reports,  articles, 
and  texts  produced  in  the  fields  of  systems,  operations  research,  economic 
analysis,  educational  analysis,  educational  evaluation,  and  cost-effective¬ 
ness  analysis.  None  require  expertise  in  any  particular  discipline  and,  as 
such,  are  well  suited  to  a  wide  range  of  reader  interests  and  needs. 

Braby,  R. ,  Henry  J.  M.  Parish,  Jr.,  W.  F.,  &  Swope,  W.  M.  A  technique 
for  choosing  cost-effective  instructional  delivery  systems.  TAEG  Report 
No.  16.  Orlando:  Training  Analysis  and  Evaluation  Group,  April  1975. 

Most  cost-effectiveness  evaluations  typically  consider  either  alterna¬ 
tive  outcomes  or  costs.  This  report  is  concerned  with  developing  a  strategy 
for  optimizing  alternatives.  It  provides  a  set  of  guidelines  for  predicting 
cost-effective  instructional  delivery  systems.  These  guidelines  are  just 
that,  they  require  technical  knowledge  on  the  part  of  the  user.  The  tech¬ 
nique  requires  that  the  user  (1)  classify  training  objectives  by  character¬ 
istics,  (2)  identify  instructional  delivery  systems  that  can  be  used  in 
meeting  common  objectives,  (3)  estimate  the  cost  of  these  systems,  and 
(4)  select  the  optimum  mix  of  systems.  Included  Is  a  FORTRAN  cost  model 
program  that  can  be  used  for  determining  component  and  system  costs. 

Defense  Economics  Analysis  Council,  Handbook  Committee,  Ecnomlc  analysis 
handbook.  Washington,  D.  C.i  U.  S.  Department  of  Defense,  2nd  Edition, 
undated. 

The  handbook  presents  the  concepts  and  methodology  used  in  economic 
analysis.  The  uninitiated  reader  is  given  a  set  of  procedures  as  a  guide 
through  the  major  phases  of  economic  analysis.  These  include:  (1)  a 
statement  of  objectives  and  presentation  of  alternative  ways  to  meet 
the  objectives,  (2)  determination  and  comparison  of  costs  and  benefits, 
and  (3)  use  of  sensitivity  analysis  in  evaluating  results.  A  description 
of  several  costing  methods  and  explanation  of  such  relevant  concepts 
as  "economic  life"  and  "discounting"  in  relation  to  DoD  policies  are 
also  provided. 

Department  of  the  Navy,  SECNAV  Instruction  7000. 14A:  Economic  Analysis 
and  Program  Evaluation  for  Navy  Resource  Management.  Washington,  D.  C. : 
Office  of  the  Secretary,  14  March  1973. 

This  instruction  provides  the  basic  guidelines  for  conducting  economic 
analysis,  including  cost-effectiveness  studies.  Included  in  the  analysis 
are  statements  of  objectives  and  alternatives.  The  major  cost  categories 
are  defined,  including  inflation  and  economic  life.  How  to  handle  output 
data  is  discussed  and  certain  types  of  output  analysis  are  explained.  A 
series  of  format  outlines  and  definitions  of  relevant  economic  terms 
are  provided. 
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Doughty,  P.  L.,  &  Stakenas,  R.  G.  An  Analysis  of  Costs  and  Effectiveness 
of  an  Individualized  Subject  Offering.  In  Sabine,  D.  (Ed.),  Accountability: 
Systems  Planning  in  Education.  Homewood,  Illinois:  ETC  Publications, 

1973,  165-191. 

A  comparison  of  four  instructional  approaches  (including  lecture-lab 
and  individualized  audio-tutorial)  in  a  university  setting  was  used  as 
the  basis  for  validating  a  cost-effectiveness  model.  The  cost  model  is 
provided,  along  with  a  statement  of  underlying  assumptions  and  guidelines 
which  may  also  be  relevant  in  other  educational  applications.  Included 
are  the  methodological  and  conceptual  steps  that  are  needed  to  develop 
a  data  base  for  this  example. 

English  J.  (Ed.).  Cost  effectiveness:  The  economic  evaluation  of  engineered 
systems.  New  York:  John  Wiley  and  Sons,  Inc.,  1968. 

This  book  is  oriented  toward  military  weapon  systems  and  avionics, 
but  many  of  the  principles  apply  to  any  cost-effectiveness  study.  In  par¬ 
ticular,  one  chapter  (A  Standardized  Approach  to  Cost-Effectiveness  Evalua¬ 
tions)  provides  specific  approaches  for  conducting  cost-effectiveness 
evaluations  where  each  step  is  described  in  detail.  Examples  dealing 
with  space  programs  and  military  weapons  systems  are  provided.  Another 
chapter  (Cost-Effectiveness  Fallacies  and  Misconceptions  Revisited) 
explains  what  not  to  do  and  how  to  avoid  common  pitfalls  in  evaluations. 

Other  useful  concepts  covered  in  additional  chapters  include  trends  in 
analysis,  basic  decision  theory,  and  analysis  of  the  systems  approach. 
Appendices  include  a  cost-effectiveness  bibliography  and  a  list  of  poten¬ 
tially  useful  criteria  for  both  cost  and  effectiveness  categories. 

Fisher,  G.  H.  Cost  considerations  in  systems  analysis.  New  York:  American 
Elsevier,  Inc.,  1971. 

This  well  written  book  addresses  specific  cost  analysis  issues  in 
a  total  systems  analysis  framework.  The  Defense  Department  sponsored 
this  Rand  Corporation  endeavor  in  order  to  assist  in  the  training  of 
national  security  analysts.  Concepts  and  principles  of  systems  analysis 
and  cost  analysis  are  defined  in  the  first  three  chapters  while  more 
specific  applications,  examples,  and  military-oriented  problems  are 
discussed  in  the  remaining  six  chapters.  Although  the  text  focuses  on 
cost  considerations  in  systems  analysis,  effectiveness  or  outcomes  cri¬ 
teria  are  incorporated  in  the  total  systems  analysis  context. 

Petruschell,  R.  L.,  &  Carpenter,  P.  MODIA  applied  in  the  design  and  cost 
analysis  of  an  innovative  Air  Force  course.  R-1021-PR,  December  1972, 

RanJ  Corporation,  Contract  No.  F44620. 73-C. 0011,  prepared  for  United 
States  Air  Force  Project  Rand. 


This  report  describes  a  methodology  for  the  design  and  cost  analysis 
of  a  military  instructional  system.  The  methodology  is  designed  to 
provide  a  means  for  designing  instructional  systems  and  the  tools  for 
assessing  the  alternatives  in  a  timely  manner.  The  specific  example 


is  used  to  show  what  the  instructional  system  will  cost  and  how  it  is 
developed.  In  order  to  do  this,  instructional  process  and  strategy 
descriptors  are  developed,  selecting  specific  media  and  determining  proper 
instructor  utilization  are  discussed,  and  a  method  for  estimating  the 
cost  of  the  system  is  presented. 

Seiler,  K.  Introduction  to  systems  cost-effectiveness.  New  York:  Wiley- 
Interscience,  1969. 

This  relatively  brief  (108  pages)  text  was  initially  designed  for 
an  audience  with  a  basic  understanding  of  calculus  and  probability  theory. 
However,  fundamental  issues  and  concepts  are  well  defined  verbally  and 
the  mathematical  models  and  notation  present  few  barriers  to  the  uninitiated. 
Sections  in  the  book  are  organized  as  follows:  I  provides  a  broad  over¬ 
relevant  cost-related  issues;  II  provides  a  definition  (with  examples) 
of  system  cost  models;  III  defines  an  effectiveness  model  which  includes 
criteria  such  as  system  performance,  availability,  reliability,  and 
survivability;  and  IV  defines  and  critiques  various  comprehensive  cost- 
effectiveness  analysis  models.  Each  section  contains  a  supplementary 
set  of  annotated  references  to  operations  research  oriented  readings. 

Temkin,  S.  An  economic  assessment  of  Army  training  alternatives.  Research 
for  Better  Schools,  Inc.  Submitted  to  Army  Research  Institute  for  Be¬ 
havioral  and  Social  Sciences,  February  1975. 

The  Army  has  recently  adopted  a  new  training  approach.  Training  Exten¬ 
sion  Course  (TEC),  which  is  an  attempt  to  provide  individualized  training 
for  a  wide  variety  of  topics  at  the  battalion  level.  This  study  compared 
operational  and  developmental  training  costs  of  TEC  versus  operational  costs 
for  conventional  group  instruction.  A  limitation  of  this  study  is  that 
only  one  alternative  was  compared  to  the  conventional  program.  Future 
cost  implications  for  expanded  use  of  TEC  and  conventional  instruction 
are  considered.  Appendices  provide  detailed  cost  breakdowns. 
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Problems 


When  cost  accounting  principles  are  applied  to  educational  or  training 
institutions,  it  is  cormon  practice  to  allocate  construction,  operation, 
and  maintenance  costs  of  physical  facilities  to  jurisdictional  cost 
centers  within  the  institution  or  center.  The  "accuracy"  of  the  allo¬ 
cation  varies  with  the  types  of  data  available.  In  some  settings,  the 
cost  of  a  "building  and  grounds"  operation  might  be  equally  distributed 
to  all  departments  either  as  administrative  overhead  or  as  a  discrete 
item.  At  other  institutions,  more  complex  distribution  patterns  may 
be  followed. 

The  costs  of  operating  and  maintaining  physical  facilities  are  current 
costs.  Allocations  are  based  on  the  full  cost  incurred  for  a  specified 
fiscal  period.  The  costs  of  construction  or  housing,  on  the  other  hand, 
are  not  charged  to  a  jurisdictional  budget  as  the  costs  are  incurred. 

While  it  might  not  be  unreasonable  to  charge  the  full  $3,000  cost  of 
a  newly  partitioned  office  to  a  department's  annual  budget,  it  would 
certainly  be  unreasonable  to  charge  the  full  cost  of  a  $3,000,000  building 
to  the  annual  budget.  Such  a  charge  could  put  a  rather  sizeable  depart¬ 
ment  $2,850,000  "in  the  red."  In  order  to  be  fair  and  not  penalize  a 
department  for  constructing  a  new  building  or  renovating  existing  facili¬ 
ties,  cost  accountants  have  devised  means  of  depreciating  facilities 
as  an  asset  over  a  long-term  period.  Furthermore,  whatever  depreciation 
schedule  is  used,  it  can  be  applied  to  the  $3,000  improvement  as  well 
as  to  the  $3,000,000  building. 

For  institutional  cost  accounting  purposes,  the  costs  of  construction, 
operation,  and  maintenance  must  be  recognized.  Similarly,  and  to  help 
balance  the  books,  the  facilities  are  listed  as  assets  and  investments. 
Charges  to  units  to  defray  the  costs  of  operating  and  maintaining  physical 
facilities  are  accepted  as  a  fact  of  life.  However,  space,  as  facilities 
are  frequently  described,  poses  a  special  problem  to  the  analyst  attempt¬ 
ing  to  identify  costs  associated  with  instructional  programs. 

Most  studies  that  have  dealt  with  instructional  program  costs  have 
adopted  the  cost  techniques  of  the  home  institution.  That  is,  they  allo¬ 
cated  to  the  program  some  portion  of  the  overhead  charged  by  the  institu¬ 
tion  to  the  department  administering  the  program.  Typically  a  mathematical 
ratio  is  used.  For  example,  if  there  are  10  departments,  one  tenth  of 
the  Institutional  overhead  is  charged  to  a  department.  Then,  if  five 
of  the  department's  fifteen  faculty  members  work  on  the  program,  one 
third  of  the  original  10%  overhead  might  be  charged  to  the  program. 

There  are,  of  course,  any  number  of  allocation  formula  variations  that 
might  be  used.  Student  credit  hours  produced,  faculty  salaries,  full-time 
equivalent  (FTE)  enrollment,  and  FTE  staff  are  more  common  bases  that 


might  be  used  in  space  allocation  formulas.  The  problem  that  quite 
clearly  arises  is  just  how  accurate  the  resultant  program  costs  are. 

If  the  program  is  composed  of  field  exercises  or  on-the-job  training, 
it  is  immediately  obvious  that  the  program  makes  far  less  demands  on 
institutional  space  than,  say,  a  biweekly  lecture  class  for  50  trainees. 

There  is  another  caveat  to  consider  when  attempting  to  allocate  the 
costs  of  physical  facilities  to  programs  on  the  basis  of  institutional 
costs  defined  by  accountants.  Not  only  is  it  difficult  to  accurately 
allocate  the  institutionally  budgeted  costs,  but  also  those  very  costs 
are  likely  to  be  inaccurate  in  terms  of  actual  out-of-pocket  costs. 

The  reason  for  this,  of  course,  can  be  found  in  the  "tricks"  of  accounting 
associated  with  asset  depreciation  and  the  problem  of  differences  between 
historical  costs  and  the  current  value  of  the  measuring  unit  ...  the 
dollar. 

Buildings  are  often  depreciated  at  a  predetermined  rate.  Regard¬ 
less  of  what  is  being,  or  has  been  paid,  on  the  principal  and  interest 
resulting  from  the  construction  or  purchase  of  the  building,  it  is  the 
depreciation  rate  that  shows.  The  actual  payments  are,  or  were,  likely 
"buried"  in  some  other  account.  In  reality,  the  depreciation  is  a  fic¬ 
tional  figure.  It  exists  only  because  some  means  is  necessary  to  recog¬ 
nize  the  possibility  of  loss  in  volume  in  an  investment.  Of  course, 
frequently  there  is  an  appreciation  in  value,  but  the  use  of  replacement 
costs  in  accounting  records  is  a  violation  of  the  historical  cost  concept 
and  of  the  necessary  assumption  that  the  dollar  is  a  stable  unit  of  value. 
Furthermore,  there  are  no  reliable  techniques  for  measuring  the  current 
costs  of  replacing  assets. 

Alternatives 

As  cost  studies  have  focused  more  and  more  on  program-level  costs, 
there  is  growing  concern  about  how  accurate  these  costs  really  are. 

There  has  been  a  growing  realization  that  it  is  not  necessary  to  adopt 
the  institution's  accounting  procedures  when  costing  a  program,  although 
that  remains  an  option.  Other  bases  which  could  be  used  to  allocate 
space  costs  are  to  compute  standard  costs,  rental  costs,  no  cost  at  all, 
or  to  compute  only  costs  that  can  be  specifically  related  to  the  existence 
of  the  program.  Each  of  these  alternatives  will  be  addressed. 

A  standard  cost  is  defined  as  "the  cost  that  should  be  incurred  to 
produce  a  given  product  or  to  perform  a  particular  operation  under  rela¬ 
tively  ideal  conditions  (Meigs,  Mosich,  and  Johnson,  1972,  p.  781)." 

In  reality  standard  costs  may  be  averages  or  reasonable  estimates  by 
experts.  The  primary  advantage  of  using  standard  costs  for  space  is 
that  all  departments  would  be  subject  to  identical  unit  costs  for  space 
of  a  given  type.  The  variable  would  be  the  amount  and  type  of  space 
needed.  The  disadvantage  of  using  standard  costs  is  that  they  imply 
an  accuracy  which  does  not  exist.  If  a  40-year-old  building  requires 
maintenance  amounting  to  $10,000  annually  while  an  average  or  ideal 
maintenance  cost  for  comparable  space  is  $7,000,  there  is  no  way  that 


the  $7,000  is  a  realistic  figure.  Standard  costs  do  tend  to  place  the 
costs  for  physical  facilities  in  perspective;  but,  desirable  as  that 
may  be,  the  use  of  standard  costs  can  give  misleading  data. 

The  use  of  standard  cost  figures  can  be  defended,  however,  when  the 
intent  is  to  export  a  particular  program  to  other  settings.  Haggart 
(1971)  illustrates  the  proper  use  of  standard  costs  when  she  couples 
cost  figures  with  the  concept  of  incremental  and  replication  costs. 

She  reports  average  costs  for  a  particular  program  and  provides  explicit 
information  about  unit  resource  requirements.  An  institution  desiring 
to  adopt  the  program  first  determines  the  cost  of  replicating  the  program 
per  resource  unit  and  then  the  incremental  units  required  to  meet  spe¬ 
cified  needs. 

Use  of  rental  costs  has  advantages  and  disadvantages  similar  to  the 
use  of  standard  costs.  Unless  the  space  is  actually  rented,  the  rental 
cost  is  someone's  estimate;  it  is  not  an  actual  cost.  The  use  of  rental 
costs  as  a  base  for  establishing  program  costs  loses  more  credibility 
when  one  considers  a  nuclear  physics  program  that  requires  a  cyclotron 
or  a  training  program  which  uses  a  large  simulator.  Rental  costs  might 
be  a  more  appropriate  base  than  standard  costs  since  unique  needs  are 
more  likely  to  be  considered,  but  the  final  result  will  remain  a  mere 
reflection  of  the  actual  costs. 

Since  the  use  of  institutional  procedures,  standards,  and  rental 
costs  are  not  likely  to  give  accurate  cost  data,  the  possibility  of  not 
allocating  facilities  costs  at  all  should  be  considered. 

There  is  a  certain  amount  of  appeal  to  not  allocating  costs  of  space 
to  instructional  programs.  Space  is,  in  many  instances,  an  incidental 
adjunct.  The  instruction  could  just  as  well  occur  in  a  tent,  on  the 
fantail  of  a  ship,  or  (in  milder  climes)  under  a  tree.  Not  costing  space 
would  have  the  same  relative  effect  as  using  standard  costs,  that  is, 
all  instructional  programs  would  be  equal  with  regard  to  what  is  rot 
always  a  bona  fide  cost  of  instruction.  The  problem  here  is  that,  in 
many  instances,  physical  facilities  are  important  to  instruction.  Pre¬ 
viously  cited  examples  of  the  cyclotron  and  the  simulator  are  more  extreme 
examples  of  instructionally  relevant  facilities.  A  minicomputer  or  a 
simple  movie  screen  are  less  extreme  examples.  If  the  physical  space 
is  integrally  related  to  Instruction,  it  seems  reasonable  that  it  should 
be  costed  to  the  educational  program. 

Costing  facilities  specifically  required  for  an  educational  program 
does  present  problems.  A  decision  must  be  made  about  what  is  required 
and  what  is  not.  Should  the  floor  space  be  costed?  What  about  the  chalk¬ 
board?  There  is  also  the  problem  of  establishing  a  "fair"  cost.  In 
his  study  of  costs  of  an  individualized  geology  course.  Doughty  (1972) 
encountered  the  problem  of  costing  resources  used  by  the  program.  Rather 
than  cost  the  space  used.  Doughty  costed  the  space  that  existed  only 
because  the  program  existed,  that  is,  actual  investment  in  and  modifica¬ 
tion  of  existing  facilities  was  charged  to  the  program.  Such  academic 


problems  as  determining  the  cost  of  inherited  space  was  thus  avoided 
and  a  more  realistic  picture  of  actual  expenditures  was  obtained. 


Several  assumptions  underlie  this  approach  to  costings.  One  must 
assume  that  (1)  all  programs  require  certain  minimal  space  (e.g.),  a 
room  which  is  supplied  by  the  institution  through  overhead  charges  to 
the  department,  (2)  these  equal  needs  balance  out  in  any  comparison, 
and  (3)  the  instructor  is  not  to  blame  if  he  is  placed  in  an  old  building 
requiring  considerable  upkeep.  Thus,  the  only  space  costs  that  are  re¬ 
cognized  as  relevant  are  modifications  to  space  for  the  benefit  of  the 
program.  In  Doughty's  study  the  modification  was  in  the  form  of  indepen¬ 
dent  study  carrels,  wall  removal,  electrical  renovation,  and  the  like. 

For  an  accountant  this  approach  to  costing  is  an  anathema.  However, 
for  a  program  administrator,  it  represents  a  realistic  picture  of  what 
a  particular  program  is  costing  in  terms  of  physical  space.  The  problem 
is  not,  after  all,  a  bookkeeping  problem  in  which  the  books  must  balance. 
Rather  it  is  one  of  determining  actual  cash  outlay  for  a  particular 
program.  To  solve  that  problem,  an  approach  is  needed  that  is  indepen¬ 
dent  of  the  stylistic,  accounting-oriented  approach. 


\ 
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Problems 


Salaries  and  wages  for  instructional  personnel  are  typically  the 
largest  components  of  an  education  or  training  program’s  budget.  Deter¬ 
mining  institutional  budgets  via  conventional  cost  accounting  techniques 
when  there  is  no  question  about  the  accuracy  of  salaries  and  wages  listed 
as  expenditures  allows  the  entire  amount  to  be  used.  However,  when 
attempting  to  allocate  a  salary  or  wage  to  specific  programs,  depart¬ 
ments,  or  disciplines  within  an  institution,  the  question  of  accuracy 
does  arise.  Short  of  actual  time  and  motion  studies,  there  is  no  precise 
way  of  allocating  personnel  time  to  specific  programs.  Even  in  the  un¬ 
likely  event  of  a  costly  time  and  motion  study,  the  data  obtained  would 
be  historical  and  the  problem  of  joint  (multifunction)  costs  would  still 
exist.  The  nature  of  education  and  training  is  so  fluid  that  staffing 
patterns  are  almost  never  constant.  Therefore,  a  time  and  motion  study 
could  provide  fairly  accurate  descriptive  data,  but  the  data  would  not 
be  reliable  enough  to  estimate  future  costs  with  the  degree  of  accuracy 
required  by  a  bookkeeper  or  accountant. 

In  spite  of  the  imprecise  nature  of  allocating  instructional  personnel 
time  to  specific  programs,  the  magnitude  of  this  cost  component  makes 
it  imperative  that  the  task  be  performed.  The  following  options  are 
articulated  for  higher  education  but  may  serve  as  useful  models  for  Navy 
education  and  training  contexts. 

Options 

At  least  four  different  methods  can  be  used  to  assign  personnel  costs 
to  programs,  departments,  or  disciplines.  Faculty  and  staff  appointment 
is  one  such  method.  A  faculty  member  may  have  a  quarter-time  appointment 
in  one  department,  a  half-time  appointment  in  another,  and  a  quarter-time 
appointment  with  a  research  project.  Using  faculty  or  staff  assignments 
to  allocate  costs  is  administratively  convenient.  The  information  is 
readily  available,  and,  as  far  as  the  institution's  books  are  concerned, 
the  method  is  accurate.  Usually,  however,  personnel  assignment  does 
not  reflect  reality.  It  would  be  a  rare  instance  in  which  faculty  ap¬ 
portioned  their  time  in  direct  proportion  to  the  assignment  ratios. 
Full-time  Navy  training  settings  are  sometimes  an  exception.  Consequently, 
while  the  method  is  a  satisfactory  accounting  procedure  on  an  institu¬ 
tional  and  jurisdictional  basis,  it  does  not  truly  reflect  program  costs 
or  even  departmental  costs  when  personnel  assignments  cross  jurisdictional 
boundaries. 

A  second  method  used  to  allocate  instructional  salary  costs  to  pro¬ 
grams,  departments,  or  disciplines  is  based  on  credit  hours  or  courses 
taught.  Theoretically,  this  method  allocates  those  costs  in  direct 
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proportion  to  expended  effort.  The  assump“ions  underlying  this  method 
of  allocation  are  that  all  credit  hours  require  equivalent  amounts  of 
involvement  on  the  part  of  specified  personnel  and  that  all  efforts  can 
be  tied  to  credit  hours.  Except  in  those  instances  where  faculty  or 
training  staff  are  assigned  full  time  to  a  specified  program  (course  , 
the  assumptions  are  not  seriously  defended.  It  is  understood,  for  ex¬ 
ample,  that  faculty  may  spend  consiaeraoly  more  time  on  a  course  which 
has  never  been  taught  before  than  on  courses  with  which  they  are  familiar. 
Furthermore,  it  is  not  always  possible  t)  attach  committee  work,  research, 
and  other  responsibilities  connected  witn  faculty  positions  to  credit 
hours. 

Despite  the  unrealistic  assumption  underlying  the  method,  allocation 
of  costs  based  on  credit  hours  taught  is  a  common  procedure.  Of  the 
19  post-secondary  education  cost  studies  surveyed  by  Doughty  and  Beilby 
(1974),  six  used  or  recommended  allocation  of  salaries  based  on  the  number 
of  credit  hours  produced.  The  arithmetic  ease  of  working  with  credit 
hours,  in  addition  to  the  availability  of  such  data,  helps  explain  why 
the  method  is  so  widely  used. 

A  third  method  of  allocating  faculty  time  employs  the  use  of  the 
staff  contact  hour.  The  staff  contact  hour  is  defined  as  the  number 
of  hours  "taught"  times  the  enrollment  (the  term  "taught"  may  be  flexibly 
applied  to  include  individual  instruction  or  self-instruction).  The 
rationale  behind  the  staff  contact  hour  is  that  the  credit  hours  assigned 
to  courses  containing  "noncredit"  labs  or  seminars  do  not  adequately 
reflect  faculty/staff  effort.  Furthermore,  in  order  to  determine  contact 
hours,  data  describing  instruction  types,  class  sizes,  faculty  work  load, 
and  faculty  needs  must  be  collected. 

The  main  argument  against  the  use  of  the  staff  contact  hour  is  that 
the  increased  accuracy  concerning  staff  requirements  and  effort  is  not 
usually  worth  the  considerable  time  required  for  data  collection  and 
computations.  Another  argument  against  its  use  is  that  it  only  considers 
time  directly  related  to  formal  courses.  Much  staff  time  may  be  devoted 
to  committee  work,  student  advisement,  research,  and  miscellaneous  plan¬ 
ning.  This  time  cannot  be  accounted  for  in  any  of  the  allocation  methods 
discussed  thus  far. 

The  fourth  method  used  to  allocate  personnel  costs  is  through  oral/ 
written  reports  from  either  the  individual  whose  time  is  to  be  allocated 
or  from  the  individual's  administrative  superior.  Only  three  of  the 
studies  surveyed  by  Doughty  and  Beilby  used  this  approach.  The  weaknesses 
of  oral  reporting  are  obvious.  The  person's  ability  to  report  accurate 
data  is  limited  by  memory,  and  Is  influenced  by  (1)  the  importance  they 
attach  to  specific  tasks,  and  (2)  the  announced  or  suspected  use  to  which 
the  data  will  be  put.  I f  an  individual  is  being  asked  to  report  on  a 
third  person's  time,  additional  variance  such  as  feelings  about  the 
person,  or  frequency  of  meetings  with  the  person,  creeps  in.  As  an 
accounting  technique,  the  "oral  report"  method  is  clumsy  and  not  very 
accurate  for  reporting  costs  that  can  be  j urisdictionally  defined.  One 


might  expect  to  account  for  anything  from  say  70%  to  130%  of  a  person's 
time.  Balancing  the  books  in  the  light  of  such  findings  could  be  quite 
a  trick!  Furthermore,  the  data  are  not  conveniently  obtained.  Expensive, 
time-consuming  interviews,  diaries,  or  questionnaire  analyses  are  required. 

Two  advantages  of  the  "oral  report"  method  of  allocation  are  that 
it  is  less  artificial  than  other  methods  and  may  be  more  accurate. 

Although,  as  previously  noted,  the  "oral  report"  method  may  account 
(hypothetically)  for  anywhere  between  70%  and  130%  of  a  staff  member's 
time,  that  does  not  mean  the  recorded  time  spent  is  in  error.  Such 
figures  may  occur  because  of  some  arbitrary  statement  about  the  rate 
of  pay  based  on  annual  salary.  As  bits  and  pieces  of  time  are  collected 
according  to  staff  reports  about  their  involvement  in  specific  tasks 
or  programs,  more  or  less  total  time  may  be  accounted  for  than  was  estimated 
to  have  been  spent.  In  other  words,  the  arbitrary  statement  about  rate 
of  pay  is  more  likely  to  be  in  error  than  staff  statements  about  how 
they  spend  their  time.  Admittedly,  it  is  unlikely  that  data  from  oral 
reports  will  be  totally  accurate,  but  it  is  likely  to  be  more  accurate 
and  useful  than  such  proxies  as  credit  hours  produced,  FTE  staff  contact 
hours,  or  administrative  assigment.  The  neatness,  convenience,  ready 
availability,  and  frequency  of  use  of  these  other  measures  for  insti¬ 
tutional  accounting  give  them  an  aura  of  credibility  that  is  unwarranted 
in  fact. 

Navy  Manpower  Billet  Costs 

Considerable  assistance  in  obtaining  total  Navy  personnel  costs  is 
provided  by  the  Personnel  Plans  Division  of  the  Bureau  of  Naval  Personnel. 
The  Division  occasionally  publishes  a  report  and  maintains  a  current 
computerized  file  that  includes  annual  billet  costs  for  Navy  officer 
and  enlisted  personnel.  These  comprehensive  cost  (to  the  Navy  and  the 
Department  of  Defense)  figures  include  the  current  and  projected  costs 
(retirement,  etc.)  of  manning  an  established  operational  military  billet 
or  a  military  billet  which  would  be  established  under  a  proposed  alterna¬ 
tive. 

These  figures  differ  considerably  from  organization-oriented  budget 
data  since  the  latter  do  not  include  such  costs  as  retirement,  transpor¬ 
tation,  and  training.  One  option  for  analysts  is  to  report  both  billet 
and  annual  salary  costs  when  comparing  instructional  alternatives. 

Obviously,  a  high  labor-intensive  option  will  fare  much  worse  when  billet 
costs  are  used  as  the  base  than  when  significantly  lower  annual  salaries 
are  reported. 
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I.  INTRODUCTION 

A.  Military  Problem 

In  an  effort  to  achieve  more  dynamic  training  in  combat  arms 
units,  the  Chief  of  Staff  of  the  Army,  on  30  June  1971,  directed  the 
decentrallzaton  of  training  management  to  battalion  level  and  lower. 

To  help  unit  commanders  carry  out  this  increased  responsibility,  the  Board 
for  Dynamic  Training  was  established  at  Fort  Benning,  Georgia.  During 
its  existence,  the  Board  identified  several  critical  problem  areas  and 
suggested  potential  solutions.  The  task  of  implementing  these  solutions 
was  assigned  to  the  Combat  Arms  Training  Board  (CATB) ,  which  succeeded 
the  Board  for  Dynamic  Training. 

One  problem  area  was  the  lack  of  a  link  between  the  service 
schools,  with  their  enormous  content  and  training  expertise,  and  the  units, 
with  their  increased  responsibility  to  conduct  training.  Unit  commanders 
were  confronted  with  the  problem  of  maintaining  individual  Military  Occupa¬ 
tional  Specialty  (MOS)  competence  as  a  prerequisite  to  developing  and 
conducting  challenging  unit  training.  Without  a  minimum  level  of  indivi¬ 
dual  skills  unit  training  was  not  profitable;  however,  the  requirement 
to  train  for  maintaining  unit  skills  was  virtually  monopolizing  the  com¬ 
manders'  attention  and  pre-empting  his  use  of  available  time  for  training 
individual  skills. 

The  Board  for  Dynamic  Training  identified  as  a  major  initial 
step  the  development  of  carefully  engineered  training  materials,  which 
would  be  prepared  by  service  schools  and  used  within  units  in  an  audio¬ 
visual  mode.  To  be  known  as  Training  Extension  Courses  (TEC),  these 
materials  would  enable  the  commander  to  maintain  individual  proficiency 
in  selected  subject  areas  with  a  minimum  of  time  and  effort  expended  at 
the  unit  level. 

B.  Current  Training  System 

At  present,  training  in  individual  skills  at  the  unit  level 
is  carried  out  by  unit  officers  and  NCO's.  Usually,  the  designated  unit 
instructor  develops  a  custom-made  program  of  instruction  (POI)  using 
available  training  literature  (which  is  often  outdated),  and  whatever 
training  aids  are  available.  The  class  is  given  and  the  POI  is  seldom 
used  again.  Little  diagnostic  testing  on  post-instruction  performance 
testing  is  ever  planned  or  conducted. 

C.  Proposed  Alternative 

The  Training  Extension  Course  (TEC)  Training  System  is  designed 
to  assist  individual  soldiers  and  unit  commanders  in  upgrading  MOS/ job 
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proficiency.  It  consists  of  multimedia  instructional  materials  prepared 
by  service  schools  for  use  in  the  field.  The  prepackaged,  soldier-valida¬ 
ted  lessons  are  interacting  and  performance-oriented.  The  TEC  Training 
System  is  designed  to  improve  individual  training  of  soldiers  in  units. 

Behind  the  TEC  program  is  the  basic  proposition  that  Army  service 
schools  will  provide  training  assistance  to  soldiers  in  field  units,  both 
Active  Army  and  Reserve  Components.  This  assistance  is  an  important 
mission  of  the  schools  and  should  be  provided  on  a  continuing  basis. 

Unit  trainers  can  use  TEC  materials  to  conduct  group  instruction 
to  train  squads,  platoons,  companies,  etc.,  in  individual  skills.  The 
TEC  system  can  also  be  used  by  individual  soldiers  in  a  self-paced  mode 
for  study  at  any  time. 

Each  of  the  lessons  sent  to  the  field  is  designed  for  a  specific 
segment  of  soldiers.  Some  lessons  apply  to  all  soldiers,  while  others 
are  appropriate  for  a  particular  branch  of  MOS. 

Before  a  lesson  is  put  into  final  form  for  distribution  to  units, 
it  is  tried  out  on  soldiers.  This  is  a  critical  step  and  represents  a 
dramatic  improvement  in  the  development  of  Array  training  materials.  At 
this  point,  the  lesson  must  bring  the  soldier  to  a  specified  standard 
of  performance.  If  it  fails  to  do  this,  it  is  rewritten  and  retested 
as  often  as  necessary  to  guarantee  that  it  will  teach  effectively. 

Another  important  improvement  found  in  TEC  lessons  is  that  the 
soldier  must  actively  participate  in  the  training.  Each  lesson  requires 
the  soldier  to  demonstrate  he  can  Jo  each  step  in  a  procedure  before  moving 
on  to  the  next  step.  This  orientation  on  performance,  as  well  as  the 
field  validation,  guarantees  TEC  lessons  will  be  effective — provided  they 
are  used. 


D.  Current  Versus  Alternative  System 

Some  principal  differences  between  the  TEC  and  conventional 
Army  training  are: 

1.  TEC  training  materials  are  designed  to  emphasize  performance — 
what  the  soldier  will  do  as  a  result  of  the  training.  The  instruction 

is  developed  and  validated  through  empirical  methods. 

2.  Unit  commanders  and  trainers  can  use  TEC  diagnostic  tests 
and  evaluations  to  determine  strengths  and  weaknesses  in  the  job  profi¬ 
ciency  of  individual  soldiers.  With  this  profile  of  strengths  and  weak¬ 
nesses  of  his  unit  as  a  whole,  the  commander  can  take  a  prescriptive 
approach  to  training. 

3.  The  TEC  training  materials  and  hardware  provide  the  nucleus 
of  a  Battalion  Learning  Center.  This  center  can  be  a  composite  of  military 
training  materials,  GKD  materials,  college  materials,  and  commerically 
produced  materials  related  to  soldier  needs  and  interests. 
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4.  Training  in  units  is  enhanced  because  the  TEC  Training  System 
provides  trainers  with  high-quality  instruction  in  a  ready-to-use  form  that 
reduces  the  need  for  unit  trainers  to  develop  lesson  plans  and  visit  training 
aids  cencers,  audiovisual  support  centers,  post  photo  labs,  or  other  agencies 
to  obtain  training  materials.  Trainers  are  thus  permitted  more  time  to 
concentrate  on  training  rather  than  on  platform  presentations. 

5.  Most  important,  before  going  to  units,  these  materials  are 
actually  "tried  out"  on  soldiers  from  the  target  population.  This  valida¬ 
tion  process  is  one  of  the  more  important  differences  between  empirically- 
designed  training  and  conventional  training.  Because  training  materials 
are  validated,  trainers  will  have  greater  confidence  that  the  training  is 
efficient  and  effective. 

II.  EFFECTIVENESS  ANALYSIS 

A.  Effectiveness  Comparisons 

1.  The  following  information  was  extracted  from  a  Cost  and  Opera¬ 
tional  Effectiveness  Analysis  Study  conducted  by  the  U.  S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  and  submitted  to  USACATB 

on  17  March  1975.  This  study  was  initiated  by  DA  DCSPER  in  the  summer  of 
1974  in  order  to  critically  evaluate  the  use  of  the  Training  Extension  Course 
(TEC)  at  the  unit  level  and  to  provide  information  to  OSD  for  budgeting 
the  TEC  program.  The  full  146-page  report  may  be  secured  from  the  U.  S. 

Army  Combat  Arms  Training  Board  (AV  835-5242,  CPT  Neal). 

2.  The  study  basically  compared  three  groups  of  field  soldiers 
in  several  subject  areas.  One  group  was  pretested,  trained  by  the  TEC 
system,  and  then  given  two  posttests — a  written  test  and  a  performance 
test.  The  second  group  was  tested  identically,  except  they  were  trained 
by  a  unit  officer  or  NCO  using  the  same  teaching  objectives  as  the  TEC 
lessons.  The  third  group  (BASELINE)  was  tested,  but  not  trained. 

3.  The  performance  measure  was  the  percent  of  test  items  correct. 

B.  Purposes  of  ARI  Cost  Effectiveness  Study 

1.  To  evaluate  training  effectiveness  of  TEC  lessons  compared 
to  conventional  training. 

2.  To  conduct  a  cost  analysis  of  TEC  training  compared  to  con¬ 
ventional  training. 

C.  Overall  Conclusion  of  the  Study 

The  conclusion  reached  from  the  study  is  that  in  both  Active 
Army  and  National  Guard  Units  the  TEC  trained  soldiers  performed  signi¬ 
ficantly  better  than  those  trained  on  conventional  methods  at  lesser  cost. 
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D. 

COMPARATIVE 

RESULTS 

. ' . '1 

(Active  Army  Only) 

LIVE 

BASE 

TEC 

INSTRUCTOR 

LINE 

Grenades 

Pretest 

46% 

42% 

51% 

Posttest 

89% 

54% 

51%  : 

Performance 

Test  74% 

43% 

45% 

LAW 

Pretest 

45% 

44% 

41% 

Posttest 

83% 

56% 

41% 

Performance 

Test  57% 

59% 

45% 

M-16 

Pretest 

43% 

38% 

39%  j 

Posttest 

78% 

48% 

39%  S 

Performance 

Test  67% 

57% 

64%  | 

81mm 

Pretest 

74% 

56% 

71  %  j 

1 

Posttest 

93% 

72% 

71%  * 

1 

Performance 

Test  72% 

41% 

26%  3 

I 

Surveyed 

Pretest 

42% 

43% 

41% 

K 

Firing 

Posttest 

67% 

57% 

41%  \ 

1 

Charts 

Performance 

Test  87% 

72% 

40%  j 

I 

NOTES: 

1 

1. 

Except  for  81mm,  all  groups  started  with 

similar  competence 

1 

levels  in 

each  subject  as  shown  by  pretest  scores 

across  groups.  j 

I 

2. 

TEC  groups  did 

significantly  better  than 

the  other  groups  in  all  ) 

I 

cases  on 

the  posttest  written  examination. 

! 

I 

3. 

Except  for  the 

LAW  and  the  M16 

rifle  groups,  TEC  groups  did 

B 

significantly  bettem  than  the  others  on 

the  performance  tests.  For  the  f 

LAW  and  M16  rifle,  TEC  and  the  live  instructor  scored  equally  well. 
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E. 

COMPARATIVE  RESULTS 

(National  Guard  Only) 

TEC 

LI 

BL 

Grenades 

Pretest 

34% 

36% 

38% 

Posttest 

90% 

38% 

38% 

LAW 

Pretest 

44% 

2% 

49% 

Posttest 

84% 

53% 

49% 

M-16 

Pretest 

51% 

47% 

41% 

Posttest 

84% 

58% 

41% 

81mm 

Pretest 

37% 

28% 

33% 

Posttest 

80% 

42% 

33% 

NOTES: 

1.  Performance  test  scores  were  not  available  in  documents  received 
by  USACATB. 

2.  The  TEC  group  scored  significantly  better  in  all  cases  than  the  other 
groups  on  the  posttest. 
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F.  SUMMARY  TO  COMPARISON 

TO 

BASELINE  GROUP 

ACTIVE  ARMY  NATIONAL  GUARD 


BL 

50% 

50% 

LI 

55% 

80% 

TEC 

80% 

95% 

NOTE:  The  BASELINE  group  mean  was  set  as  the  norm.  The  mean  of  the  BASELINE 
group  represents  the  point  where  50%  of  the  soldiers  are  below  the  comparative 
average  and  50%  are  above  the  comparative  average.  The  TEC  and  LI  groups  are 
then  compared  to  the  established  norm. 


G.  CORRELATION  BETWEEN  PERFORMANCE  TEST  AND  GT  SCORES, 

BY  LESSON  CATEGORY  AND  EXPERIMENTAL  GROUP, 

ACTIVE  ARMY 


Lesson  Category  Experimental  Group 


TEC 

LI 

BL 

Hand  Grenades 

-.02 

.61* 

.24 

LAW 

.06 

.54* 

.44* 

M16  Rifle 

.02 

.24 

.43* 

Mortar  FDC 

-.13 

.27 

.71* 

Surveyed  Firing  Charts 

.02 

.11 

.27 

*  Significantly  higher  than  zero,  p  <  .05 

NOTE:  This  table  shows  that  TEC  teaches  independently  of  GT  score,  whereas 
the  degree  of  live  instructor  group  learning  is  dependent  upon  GT  score 
(l.e..  Innate  learning  abilities). 
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III.  COST  ANALYSIS 


A.  Although  the  TEC  Cost  Effectiveness  Study  followed  the  guidelines 
set  forth  in  AR  37-13,  it  was  impossible  to  use  the  exact  formats  specified 
in  that  regulation  and  still  give  an  adequate  statement  of  the  impact 

of  the  TEC  program.  The  hardware  and  software  are  complementary;  neither 
can  stand  alone.  USACATB  has  modified  the  formats  of  AR  37-13  in  an 
attempt  to  summarize  pertinent  costs.  The  following  format  compares  the 
costs  of  conventional  training  to  TEC  training  in  operational  units  over 
a  1064  battalion  equivalent  base  of  100  hours  of  annual  instruction. 

Under  current  funding  TEC  will  not  be  fully  implemented  to  support  the 
1064  battalion-equivalent  units  until  FY  1977.  Start  up  and  initial 
distribution  costs  are  shown  from  1974. 

B.  The  TEC  system  is  not  designed  to  teach  all  individual  skills 
within  a  unit.  However,  best  estimates  indicate  that  at  least  100  hours 
of  instruction  per  man  per  battalion  can  be  expected  annually.  Thus, 
the  cost  of  providing  that  100  hours  under  both  alternatives  is  shown 

in  this  analysis. 

C.  ECONOMIC  ANALYSIS  PROGRAM  EVALUATION  SUMMARY  OF  COSTS  FOR 

FORMAT  A-l . 

1.  Submitted  DoD  Component :  TRADOC  (DA) 

2.  Date  of  Submission:  15  July  1975 

3.  Project  Title:  Training  Extension  Course 

4.  Description  of  Project  Objectives:  Provide  individualized, 
self-paced,  systems  engineered  instruction  in  individual  soldiers  job 
skills  to  soldiers  in  units. 

5a.  Present  Alternatives:  Conventional  live  instructor-oriented 
lectures,  demonstrations,  and  conference-based  instruction  in  units. 

5b.  Proposed  Alternative:  Individualized,  self-paced,  systems 
engineered  standard  instructional  packages  in  audio-visual,  audio  only,  and 
written  format . 

6a.  Economic  Life  of  Present  Alternative:  One  year  field  research 
shows  that  present  unit  instruction  is  tailor-made  for  each  instructional 
situation  and  almost  never  reused  in  subsequent  sessions. 

6b.  Economic  Life  of  Proposed  Alternative:  TEC  hardware  and  software 
will  have  an  estimated  average  economic  life  of  six  years.  The  development 
and  revision  of  software  is  continuous. 


Alternative  a.  -  Conventional  Training  in  Units 


ogram/Proj ect  Costs  ($  Million) 
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$42 . 1M  in  1975  dollars  becomes  $50,441  in  1977  dollars  due  to  inflation 


10a.  Total  Project  Costs  Discounted  Thru  FY  1980: 


Present  Alternative  $363,404 

Proposed  Alternative  (TEC)  $246,072 

10b.  Uniform  Annual  Costs; 

Present  Alternative 
Proposed  Alternative  (TEC) 

11.  Terminal  values  cannot  be  estimated. 

12.  Net  costs  cannot  be  calculated  (SAe  10  &  11) 

13.  Source  Derivation  of  Cost  Estimates: 

All  costs  were  derived  from  the  USARI  study  and  FY  74  and  FY 

75  USACATB  Comptroller  files.  A  10%  inflation  factor  was  used  for  FY 

76  thru  FY  80  and  a  10%  discount  rate  was  used  for  derivation  of  present 
values . 

14.  Project  Officer: 

CPT  William  D.  Neal 
ORSA  Project  Officer 
TEC  Division,  USACATB 
(AV  835-5242) 

D.  The  TEC  Lesson  Development  Dollar,  by  Cost  Category:  In 
Percentage 

Source:  ARI  COEA 
Average  Cost  ■  $15,920 

NOTE:  The  $15,920  represents  the  average  total  cost  to  develop 
the  prototype  TEC  answer  print. 

E.  Annual  cost  to  support  typical  combat  arms  battalion  in  conven¬ 
tional  instruction 

The  annual  costs  for  providing  an  equivalent  amount  of  conven¬ 
tional  instruction  to  a  typical  combat  arms  battalion  is:  $47,437.  There¬ 
fore,  TEC  is  $9,158  less  expensive  per  battalion  per  year  on  the  average. 
This  represents  a  cost  avoidance  in  that  TEC  requires  officer/NCO  support 
at  the  battalion  level;  thus,  these  supervisors  can  devote  more  time  to 
training  in  unit  skills  as  opposed  to  Individual  skills. 


$  51.915 
$  35.153 


C-ll 


F.  Annual  Costs  at  Full  Implementation 

Considering  all  costs — developmental  and  operational — the  costs 
to  support  100  hours  of  equivalent  instruction  per  man  per  battalion 
over  1,064  battalion  equivalent  units  annually  in  1975  dollars  is: 

TE  CONVENTIONAL 

$42,100,000  $50,473,000 

Again  this  difference  is  a  cost  avoidance  rather  than  a  cost  savings. 

IV.  NON-QUANTIFIABLE  ELEMENTS  ANALYSIS 

The  following  statement  was  extracted  from  a  paper  written  by  Dr. 
Charles  Schuller  after  two  years  of  working  in  the  TEC  program.  Dr. 

Schuller  recently  retired  as  Professor  of  Education  and  Director  of  the 
Instructional  Media  Center,  Michigan  State  University. 

SCHULLER,  12  February  1975 

Benefits  realized  from  Army  TEC  Program  aside  from  specific  cost  measur¬ 
able  benefits. 

Significant  indirect  benefits  have  been  achieved  by  the  TEC  program 
in  addition  to  the  increased  efficiency  and  effectiveness  of  the  training 
directly  involved.  Notable  among  these  are  the  following: 

1.  Throughout  the  Army  establishment  from  top  command  to  company 
commanders  and  platoon  leaders  in  the  field,  attention  has  been  redirected 
to  major  emphasis  on  the  learning  needs  of  trainees  rather  than  on  the 
teaching  needs  of  instructors.  This  shift  in  emphasis  to  the  learner 
represents  a  fundamental  change  in  Army  training  policy  and  makes  possible 
for  the  first  time  the  development  of  scientifically  based  learning  systems 
throughout  the  military  training  program.  This  development  is  a  consequence 
of  the  fact  that  the  TEC  program  initiated  the  thrust  Cowards  learner- 
centered  training. 

2.  As  a  result  of  TEC's  developmental  work,  the  Army  now  has  a 
training  research  and  development  capability  (based  in  part  on  the  CATB 
28  step  development  model)  which  can  readily  be  applied  to  other  current 
and  future  training  needs.  The  Army  did  not  have  that  capability  prior 
to  the  TEC  program — and  an  investment  of  millions  of  dollars  and  hundreds 
of  man  years  would  be  required  to  achieve  the  current  status  of  R&D  cap¬ 
ability  were  such  an  effort  just  being  initiated.  At  this  time  the  R&D 
capability  has  been  centered  at  USACATB  and  the  nine  TRADOC  Service 
Schools  in  the  TEC  II  and  TEC  III  programs.  As  the  TEC  IV  Schools  begin 
to  develop  their  material,  the  same  R&D  capability  will  be  extended. 
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3.  The  experience  of  service  school  instructional  staffs  working 
on  TEC  programs  has  helped  bring  about  closer  self-scrutiny  and  re- 
evaluation  of  what  they  were  doing  in  resident  instruction.  This  in¬ 
fluence  of  TEC  is  reflected  in  two  significant  ways:  (a)  changed  atti¬ 
tudes  on  the  part  of  instructors  with  respect  to  the  effectiveness  on  their 
own  instruction,  and  steps  necessary  to  improve  it,  and  (b)  actual  modifi¬ 
cation  of  course  programs  as  indicated  below: 

a.  At  the  U.  S.  Army  Signal  School,  Fort  Gordon,  Georgia,  TEC 
is  currently  programmed  to  replace  an  eight  week  long  resident  course — 
SWITCHBOARD  OPERATOR,  MOS  72C.  There  are  1600  jobholders  for  this  MOS 
in  field  units.  It  will  take  approximately  26-28  TEC  lessons  to  replace 
this  course.  They  will  start  development  in  June  1975,  and  be  available 
by  Fall  1976.  Additionally,  every  TEC  lesson  developed  by  the  Signal 
School  (currently  29  in  number)  will  be  converted  to  a  non-resident 
sub-course.  Conversion  time  is  minimal  and  means  more  and  better  written 
sub-courses  available  without  an  increase  in  the  number  of  course  writers 
(resources)  required.  Currently,  the  three  TEC  lessons  on  the  AN/PRC-77 
have  been  so  converted  and  have  met  with  overwhelming  success. 

The  Signal  School  is  currently  discussing  the  "conversion" 
of  TEC  lessons  for  classroom  presentation  by  a  line  instructor.  As  soon 
as  subject  areas  are  developed  as  complete  TEC  packages,  they  will  be  con¬ 
verted  and  directed  toward  resident  instruction. 

In  summary  the  Signal  School  will  realize  the  benefits  of 
TEC  not  only  through  TEC  lessons  for  field  use,  but  also  for  use  in  both 
resident  and  non-resident  instruction. 

The  Field  Artillary  School  (USAFAS)  and  Air  Defense  School 
(USAADA)  have  requested  and  received  an  additional  issue  of  75  and  60  TEC 
hardware  machines  respectively  for  use  in  resident  instruction.  These 
machines,  and  related  software,  will  be  used  to  both  supplement  and  re¬ 
place  platform  instruction.  Likewise,  the  Infantry  School  (USAIS)  has 
utilized  TEC  equipment  for  supplementary  instruction  since  July  1973. 

Comparable  indicators  from  other  schools  make  clear  that 
TEC  lessons  are  applicable  in  some  degree  to  resident  training  in  all 
schools  and  that  this  influence  is  expected  to  become  of  major  significance 
over  the  next  few  years. 

4.  Additional  Influences 


a.  TEC  has  rationalized  doctrine  at  all  schools.  This  has 
>**n  prompted  by  a  confirmation  of  high  level  reviews  of  TEC  lessons  and 
? be  specification  required  in  the  TEC  lesson  development  process. 
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EXAJtPLE  -  High  level  review  of  TEC  lessons  on  patrolling 
brought  about  change  in  doctrine  from  highly  specialized  Ranger  patrol 
techniques  to  more  general  techniques  applicable  to  non-specialized  per¬ 
sonnel  in  field  units. 

b.  TEC  has  corrected  errors  and  omissions  in  GFM's. 


EXAMPLE  -  Contractor  brought  to  Armor  School's  attention 
the  fact  that  a  critical  toggle  switch  in  the  tank  turret  was  not  iden¬ 
tified  or  dealt  with  in  any  of  the  training  or  service  manuals.  The 
omission  was  brought  to  light  through  the  learning  task  analysis  steps 
in  the  CATB  development  model. 

c.  TEC  has  stimulated  innovative  training  techniques. 

EXAMPLE  -  The  Armor  School  developed,  inhouse,  a  tape  re¬ 
corder  input  to  plug  into  the  tank  intercom  system  going  to  all  stations 
so  the  whole  crew  could  hear  and  train  as  a  group.  The  material  used 
initially  was  a  TEC  audio  visual  lesson  designed  for  single  individual; 
this  was  converted  to  audio-only  and  used  in  the  above  manner  with  highly 
successful  results. 


d.  TEC  is  helping  standardize  training  throughout  the 
system.  By  reason  of  the  care  and  precision  of  the  development  process 

and  the  fact  that  service  school's  content  experts  are  responsible  for  their 
development,  TEC  lessons  are  of  uniformly  high  quality.  That,  plus  the 
fact  that  they  are  uniquely  effective  training  materials  makes  for  their 
ready  acceptance  and  use  throughout  the  system.  This  will  result  both  in 
more  effective  and  more  standardized  instruction  wherever  given. 

e.  TEC  has  upgraded  the  capability  of  the  TRADOC  Schools 
to  manage  R&D  projects.  Being  a  totally  new  and  innovative  program  cen¬ 
tered  within  the  TRACDOC  School  system,  the  TEC  project  has  forced  managers 
within  the  schools  to  tackle  and  solve  the  typical  problems  of  resource 
allocation,  staff  reorganization,  and  project  control  that  are  necessary 
for  successful  R&D  program  management.  Prior  to  the  TEC  program,  R&D 
management  techniques  were  almost  exclusively  applied  to  hardware  systems. 
TEC  has  made  it  necessary  to  apply  the  same  techniques  to  a  training 
system.  Consequently,  the  TEC  system  has  become  a  model  of  other  training 
oriented  R&D  projects  within  TRADOC. 
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V. 


CONCLUSIONS 


A.  Relative  Effectiveness  Calculations 


Overall  Effectiveness 


Cost  ($  Million) 


TEC  87.5* 
CONVENTIONAL  67.5* 
BASELINE  50.0 


42.7 

50.5 

50.5 


*Active  Army  and  National  Guard  averaged  (Equal  N's  assumed) 

1.  Relative  effectiveness  (TEC  vs  Baseline)  =  87.5/50.0  =  1.75 

2.  Relative  effectiveness  (TEC  vs  Conventional)  *>  87.5/67  5  ■  1.30 

3.  Relative  effectiveness  (Conventional  vs  Baseline)  * 

67.5/50  0  =  1.35 


B.  Relative  Cost  Calculations 

1.  RC  (TEC  vs  Baseline)  =  42.7/50.5  =  .84 

2.  RC  (TEC  vs  Conventional)  =  42.7/50.5  =  .84 

3.  RC  (Conventional  vs  Baseline)  =  50.5/50.5  =  1.00 

C.  Relative  Worth  Calculations 

1.  RW  (TEC  vs  Baseline)  =  1.75/. 84  *  2.08 

2.  RW  (TEC  vs  Conventional)  =  1.30/. 84  =  1.54 

3.  RW  (Conventional  vs  Baseline)  m  1.35/1.00  »  1.35 

D.  Conclusions  of  the  ARI  Study 

1.  On  the  average  TEC  instruction  teaches  both  written  posttests 
and  hands-on  performance  tests  better  than  conventional,  live  instructor, 
instruction. 


2.  The  state  of  training  in  individual  skills  in  both  the  Active 
Army  and  National  Guard  is  generally  poor. 

3.  TEC  teaches  soldiers  with  low  abilities  and  high  abilities 
equally  well,  whereas  conventional  instruction  is  not  as  effective  with  low 
ability  soldiers  as  it  is  with  high  ability  soldiers.  Over  all  ability 
groups  TEC  teaches  "better." 

4.  Considering  all  costs,  TEC  is  more  cost-effective  than  con¬ 
ventional  Instruction. 
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APPENDIX  D 


NAVY  IC  SCHOOL  COST  ANALYSIS 


i 

I 


INTERIOR  COMMUNICATIONS  "A"  SCHOOL  COST  ANALYSIS 


All  of  the  physical  resource  requirements  for  implementing  the  IC  "A" 
School  have  been  broken  down  into  the  following  categories:  (1)  Personnel, 
(2)  Hardware,  (3)  Software,  and  (4)  Facilities,  Within  each  category 
several  indices  were  sought.  These  include: 

Personnel 

Number  of  students 

Average  student  salary  cost 

Number  of  instructors 

Average  instructor  salary  cost 

Instructor  pre-service  training  salary  costs 

Hardware 


New  procurement 

Replacement 

Maintenance 

Software 


Instructor  course  revision  salary  cost 
Printing 

Films,  slides,  manuals,  etc. 

Facilities 


Classroom  space 
Utilization 
Furnishings 
Maintenance 

Within  these  categories,  some  administrative  costs  are  embedded.  Whenever 
possible,  they  are  specified.  Many  of  the  costs  within  the  categories 
are  derived  indirectly  because  the  funding  comes  from  a  variety  of  sources 
that  are  not  readily  identifiable.  These  indirectly-derived  costs  are  in¬ 
dicated. 

Personnel 


Number  of  Students.  The  total  number  of  student  for  FY75  was  1,032. 
Of  these  a  sample  of  200  was  drawn  from  all  classes  to  examine  the  rate 
structure.  The  results  are  shown  below: 


Rate 

Pay  Grade 

N 

Seaman  Recruit 

E.l 

92 

Seaman  Apprentice 

E.2 

31 

Seaman 

E.  3 

65 

IC3 

E.4 

7 

IC2 

E.5 

4 

IC1 

E.  6 

1 

2>200 


Of  these,  19  receive  commuted  rations  (COMRATS)  (an  additional  food 
allowance) ,  39  have  dependents  and  receive  Basic  Allowance  for  Quarters 
(BAQ).  The  average  student  is  in  the  E-2  pay  grade  with  less  than  two 
years  service.  Of  the  51  classes  covening  in  the  last  year,  there  was 
an  average  of  21.4  students  (a  =  3.3)  in  each  class.  Of  these,  approxi¬ 
mately  five  students  (a  =  2.9)  were  set  back  during  each  cycle  for  a  total 
of  approximately  six  weeks/cycle.  That  is,  of  the  average  five  students 
each  cycle,  some  require  more  than  one  week  of  additional  training. 

Average  Annual  Student  Salary  Costs.  This  figure  could  be  derived  by 
determining  the  actual  costs;  which  are  the  basic  salary,  BAQ,  COMRATS, 
uniform  allowance,  and  travel  to  the  training  center. 


Average  student  salary  (E.2  with  <  2  yrs  service) 

BAQ  (no  dependents)* 

COMRATS* 

Uniform  Allowance 

Total  Annual  Salary  Costs/E-2 

However,  by  using  the  Navy  composite  standard 
military  rate  table,  a  comparable  figure  is  obtained. 

For  an  E.2  apprentice  the  annual  rate  is: 

or,  the  salary  per  student  per  cycle  (125.25/wk  x  8  wks)  = 


$4600.80 

810.00 

867.60 

68.40 

$6346.80 


$6514.00 

$1002.00 


Student  Salary  Costs  per  Training  Cycle 

•  Total  salaries  per  cycle 

($1002.00  x  21  students)  $21,042.00 

•  Additional  salaries  for  setbacks  per  cycle 

($125.25/wk  x  6  wks)  751.50 

•  Travel 

(103.17/student  x  21  students)  2,166. 57 

Total  Student  Costs  per  Training  Cycle  $23,959.67 


*All  students  receive  BAQ  and  COMRATS  in  cash  or  kind.  The  majority 
recieve  the  latter. 
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Using  the  same  cost  categories,  but  with  the 

total  costs  for  establishing  an  operational  billet 

taken  into  account,  the  total  student  costs  per 

training  cycle  are:  $45,819.00 

Number  of  Instructors.  There  are  presently  three  teams  of  six  instruc¬ 
tors  each  in  the  IC  "A"  School.  Each  team  has  an  E-7  supervisor/instructor. 
Their  rate  structure  is  shown  below: 


Rate 

Pay  Grade 

N 

IC2 

E-5 

2 

IC1 

E-6 

16 

ICC 

E-7 

7 

IN  -  21 

The  average  instructor  has  three  dependents  and  approximately  14  years 
(a  ■  5)  of  service. 


Average  Annual  Instructor  Salary  Costs.  The  average  instructor  is  in 
pay  grade  E.6  with  14  years  service  and  receives  both  BAQ  and  COMRATS. 


Basic  Salary  (E.6  with  15  yrs)  $8,427.60 
BAQ  (average  3  dependents)  1,328.40 
COMRATS  867.60 
Uniform  Allowance  100. 00 


Total  Annual  Salary  Cost/E-6  $10,724.40 

Again,  by  using  the  Navy  composite  standard 

military  rate  table,  a  figure  taking  these 

factors  into  account  is  obtained.  For  an  E.6 

P01,  the  annual  rate  is:  $11,788.00 

or,  the  average  salary  per  instructor  per  week  —  $  266.68 


Taking  into  account  all  billet  costs,  the 
average  instructor  costs  the  government 

$22,305.00  or  per  week  —  $  429.00 

Instructor  Salary  Costs  per  Cycle.  These  costs  may  be  specified  in 


many  forms  due  to  several  variable  costs  that  are  encountered  prior  to 
and  during  the  training  cycle.  These  include: 


•  Instructor  School 

(226. 68/wk  x  4  wks)  $  906.76 

•  Classroom  sit-in/familiarization 

(226. 68/wk  x  up  to  16  wks,  average  8  wks)  1,813.57 

•  Off-platform  duties 

(course  revision,  review  board,  etc.) 

(226. 68/wk  x  1  wk/cycle)  226.68 

•  Average  Travel  Costs/Instructor/Tour  1,668.33 

In  addition,  certain  support  costs  arising  from  the  instructor  school  could 
be  considered  as  accruing  to  training  cycle  costs. 

If  the  average  instructor  serves  a  three-year  tour  instructing,  then 
the  above  training  and  travel  costs  could  be  considered  as  being  amortized 
over  approximately  132  weeks  of  instruction,  allowing  for  leave  time. 

The  cost  to  each  cycle  for  each  instructor  becomes: 

$906.76  +  $1,813.92  +  1,688.33  =  $4,380.61/16.5  cycles  = 

$265. 98/cycle 

Instructor  costs  per  cycle  include  the  following  components: 

•  Average  instructor  salary/cycle 

226.69/wk  x  7  wks  $1,586.83 

•  Total  instructor  salaries/cycle 

$1,586.83  x  6  instructors  9,520.98 

•  Training  and  Travel  Costs 

$4, 380. 61/16.5/cycles  265.98 

Total  Instructor  Costs/Training  Cycle  $11,373.79 

If  the  additional,  off-platform  duties  are  considered  a  cost  accruing 
against  the  "A"  School,  then  the  following  additional  expense  must  be 
considered  within  each  training  cycle  budget: 

$226.69/wk  x  instructors  $  1,360.14 

+  11,373.93 

12,733.93 
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Hardware 


Acquisitions.  According  to  the  "A"  School  materials  office  administra¬ 
tor,  no  new  equipment  was  purchased  last  year.  The  present  equipment  in¬ 
cludes  the  following  items: 

1  MK  11  Gyro  Compass 
1  MK19  Master  Gyro  Compass 

1  MK27  Gyro  Compass 
8  PSM.4R  VOM 

12  PSM.4C  VOM 

2  USM. 105A  Oscilloscopes 
15  USM. 117  Oscilloscopes 

1  Megger 

13  URM  127  Audio  Signal  Generators 

1  Mechanical  Shaft  Tachometer 

2  Strobescopic  Tachometers 
2  Tube  Testers 

Several  motors  and  generators  (some  inoperable) 

Misc  safety  equipment  (shorting  bar,  etc.) 

Misc  loudspeakers  and  microphones 

1  LS-386  Intercom  System 
1  IC  Switchboard 

1  Ship's  Service  Distribution  System 

1  Fire  Alarm  Switchboard 

20  16mm  Projectors 

2  8mm  Cassette  Projectors 
25  OH  Projectors 

1  Reel-to-Reel  Tape  Recorder 

Some  of  the  minor  equipment  is  not  included  on  inventory.  The  two 
8mm  projectors  are  recent  acquisitions  that  are  "automatic"  issue  from  the 
Material  Support  Officer  (MSO). 

Replacement .  This  category  does  not  appear  to  be  particularly  appli¬ 
cable.  Almost  all  equipment  is  obsolete  at  the  time  it  is  surveyed,  thus 
it  is  typically  replaced  with  a  newer  piece  of  gear  that  more  readily 
fits  the  acquisition  category. 

All  miscellaneous,  small,  consumable  expenses  are  paid  by  the  MSO. 

To  cover  these  costs  $100/quarter  is  allocated  for  administration  and 
$1,500  each  quarter  for  the  "A"  School.  Examination  of  these  records 
shows  a  typical  expense  at  the  division  (administrative)  level  being 
approximately  $100  for  a  bookcase,  $15  for  transistors,  etc.  At  the  "A" 
School,  typical  expenses  involve  wire  and  cable  @  $125,  smocks  @  $75, 
answer  cards  @  $30,  and  a  printing  over  run  category,  batch  posting,  which 
provided  a  $77  addition  to  the  budget  due  to  a  return  of  funds  committed 
the  prior  year.  All  of  these  costa  recurred  throughout  the  last  fiscal 
year.  Naturally,  both  the  budgets  are  exhausted  at  the  end  of  the  fiscal 
year. 
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Maintenance.  Equipment  maintenance  is  handled  in  a  number  of  ways. 

Test  equipment  is  maintained  by  the  Field  Calibration  Laboratory.  This 
facility  handles  maintenance  for  all  test  equipment  on  the  NTC  compiler. 

They  can  furnish  no  exact  figures  for  repair  or  calibration,  indicating 
that  wide  variances  in  equipment  are  found.  If  necessary,  some  hourly 
estimate  can  be  derived,  but  accuracy  is  unknown. 

A  similar  problem  exists  for  audiovisual  equipment.  Some  repair 
is  carried  out  by  students  at  the  AV  Repair  School  located  on  the  NTC 
compound.  Some  are  sent  to  the  Naval  Educational  Training  Support  Center. 
Both  repair  facilities  have  no  fee  schedule. 

Maintenance  on  operational  equipment  appears  to  be  casual.  Students 
perform  minor  repairs  to  switchboards  during  training.  Every  fifth  week, 
the  instructors  rotate  "off-platform"  to  perform  a  variety  of  functions, 
including  maintenance.  However,  no  one  is  able  to  give  any  estimate  of 
the  percent  of  time  spent  performing  these  functions.  It  appears  to  be 
a  low  estimate.  Office  equipment  is  regularly  serviced.  This  is  handled 
by  the  MSO  who  spent  $210  during  the  last  FY  for  these  services. 

Present  Budget.  Both  "A"  and  "C"  Schools  spent  a  total  of  $11,259 
for  "consumable  parts  and  supplies"  in  the  last  FY  of  which  slightly 

over  half  went  to  the  "A"  School.  The  anticipated  FY76  budget  has  been 

cut;  the  total  allowance  being  $7,600.  To  implement  complete  individualiza¬ 
tion  of  the  "A"  School,  approximately  $400,000  -  $500,000  is  needed  for 
equipment  and  site  preparation.  Because  this  money  is  not  available,  CNTT, 
Memphis  has  informed  MSO  that  implementation  must  be  held  off  until  1978. 

Facilities 

Classroom  Space.  There  is  a  total  of  nine  lecture  classrooms  in  use 
by  the  "A"  School  not  including  administrative  and  instructor  office  space. 
Each  of  these  rooms  is  24'  x  31’  which  provides  a  total  of  6,696  square 

feet  of  classroom  space.  In  addition,  one  room  is  shared  by  two  weekly 

sections  for  laboratory  exercises  and  one  room  houses  an  IC  control  center 
mock-up.  These  two  rooms  are  similar  in  size  bringing  the  total  space 
for  "A"  School  to  8,184  square  feet. 

Furnishings  and  Utilization.  Each  of  the  nine  classrooms  is  furnished 
with  sufficient  student  desks  to  meet  the  needs  of  the  particular  class. 

This  can  vary  from  12  to  25.  Hue  to  the  fluctuation  in  the  number  of 
weekly  student  input,  some  shuffling  of  available  desks  is  necessary,  but 
no  funds  are  expended  for  new  desks.  A  similar  situation  prevails  for 
tables,  chairs,  etc.  The  utilization  rate  for  all  furnishings  varies  with 
each  weekly  student  input  typically  ranging  from  12  to  25  students  maximum. 
Any  additional  requirements  are  typically  dealt  with  through  unofficial 
channels. 
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Maintenance.  The  major  source  of  labor  for  routine  maintenance  con¬ 
sists  of  entering  students  in  a  "wait"  status.  As  they  normally  have  no 
other  duties  during  the  wait  period,  their  labor  is  not  considered  a  cost 
accruing  against  "A"  School  operation.  Minor  facility  replacement  costs 
have  been  extremely  difficult  to  ascertain.  The  total  NTC  maintenance 
budget  for  FY76  is  $2,210,000,  but  what  portion  is  applicable  to  the  1C 
School  is  unknown.  According  to  the  staff,  some  maintenance  of  walls  has 
been  prevented  for  several  years  now  due  to  a  lack  of  funds. 

Software 

Course  Revision.  Several  revisions  occurred  in  the  past  year.  The 
mechanism  for  initiating  these  revisions  was  a  weekly  meeting  of  approx¬ 
imately  15  instructors  that  lasted  for  1-2  hours.  Typically  the  instruc¬ 
tor  population  of  these  meetings  consisted  of  three  Chiefs  and  12  Petty 
Officers.  Based  on  the  decision  made  at  these  meetings,  course  revision 
was  accomplished  during  the  one  week  in  five  when  the  instructor  rotated  off 
platform.  The  approximate  costs  of  these  personnel  can  be  obtained  by  ex¬ 
amining  the  personnel  costs.  The  meetings  recessed  in  March  and  are  not 
scheduled  to  be  reconvened  due  to  their  impending  individualization  of  this 
course.  However,  all  instructors  still  rotate  one  week  in  five. 

Printing.  Minor  nonlnstructional  services  for  the  "A"  School  are 
handled  through  the  M SO.  This  amounted  to  $30  for  the  last  FY.  Student 
handouts  and  instructor  lesson  guides  are  handled  through  a  reimbursible 
account.  Examples  of  typical  costs  that  have  occurred  during  the  last  FY 
are  given  below.  These  figures  include  plates  which  could  be  reused,  but 
due  to  frequent  revision  in  the  past,  have  usually  been  discarded. 


Total  Copies 

No. 

of  Pages 

Length  of  Use 

One  Side 

Two  Sides 

Cost 

1 

2000 

85 

12 

months 

X 

$1,127 

2 

1000 

30 

12 

months 

X 

$  340 

3 

2000 

34 

12 

months 

X 

$  416 

4 

1750 

8 

12 

months 

X 

$  87 

5 

1000 

8 

12 

months 

X 

$  55 

TOTAL  7750 

$2,025 

3000  one 

side 

x  115  - 

3.45  x 

103  pp  “ 

.00425/p 

$1,467 

4750  two 

sides 

i  x  50  - 

2,375  x  103  pp  - 

.0023/p 

558 

Another  potential  source  of  prining  costs  arises  from  use  of  the  Xerox. 
This  is  difficult  to  specify  quantitatively,  but  it  appears  that  some 
training  material  is  reproduced  from  time  to  time  on  the  copier. 
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Manuals  and  Films.  These  costs  are  minor.  Films  seem  to  spring  from 
the  earth  and  are  rarely  replaced.  Transparencies  cost  $.35  each  and  are 
rarely  replaced.  Technical  manuals  are  purchased  through  the  Naval  Educa¬ 
tion  Training  and  Development  Center  (NF,TDC)  at  unknown  costs.  An  approx¬ 
imate  figure  of  $5.00  per  manual  along  with  a  35%  replacement  rate  is 
assumed. 


Personnel  Costs  FY75 
Interior  Communications  "A"  School 
Naval  Training  Center,  San  Diego,  California 


COST 

SUMMARY 

Coat  Category 

Billet  Cost  (in  $1 

Salary  Cost  (In  $) 

Annum 

Cycle 

Annum 

Cycle 

Instructor  Salary 

Instructor  Training  Salary 
Administrat ion/Mlsc  Salary 
Student  Salary 

6,13} 

849 

6,073 

35,822 

312,761 

4  1,281 
309,759 
1,826,914 

3,2  >4 
447 
3,213 
20,563 

164, yn 
22,798 
163,885 
1,048,689 

TOTAL 

48,877 

2,492,715 

27,457 

1,400,285 

These  data  are  based  on  the  following  information: 

1.  An  eight  week  training  cycle. 

2.  A  total  of  51  training  cycles  during  FY75. 

3.  An  additional  6  student  weeks  per  cycle  for  setbacks  and  additional 
training. 

4.  An  average  of  approximately  20.2  students  per  cycle. 

5.  A  total  of  1032  students  during  FY75. 

6.  A  total  of  21  instructors  and  supervisors  (providing  the  equivalent 
of  14  full  time  instructors). 

7.  A  total  of  7  administrative  personnel  (including  the  instructors 
administrative  tire,  there  is  the  equivalent  of  14  full  time  administra¬ 
tors)  . 
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Trainin; 


cles  and  Student  Training  Time 


8  Cy 

There  are  51  cycles  per  year,  with  a  new  cycle  beginning  every  Monday 
(except  Christmas  week) .  A  week  of  "Electrical  Gyros"  is  offered  during 
every  cycle  and  a  week  of  "Mechanical  Gyro"  is  offered  every  other  cycle. 
When  both  gyro  classes  are  offered  in  a  single  cycle,  they  are  conducted 
concurrently  by  different  instructors.  The  majority  of  students  need  only 
one  or  the  other;  thus  by  rearranging  schedules  the  necessary  training  is 
readily  provided.  Students  which  require  only  electrical  gyros  progress 
through  eight  sequential  weeks.  Students  receiving  instruction  in  mech¬ 
anical  gryos  may  or  may  not  sequence  through  in  the  same  fashion  as  students 
in  the  electrical  gyro  sequence.  However,  the  end  result  is  the  same; 
both  groups  receive  eight  weeks  of  training.  As  some  students  require 
both  weeks  of  gyro  training,  these  students  may  be  considered  as  "set  back" 


an  additional  week  which  increases  student  training  time, 
meters  are: 

The  cycle  para' 

1. 

Basic  number  of  student  training  weeks /cycle 
(20.2  students/cycle  x  8  weeks)  ■  161.6 

2. 

Number  of  additional  student  weeks  due  to  setbacks 
gyro  training/ cycle  ”  6 

and  additional 

3. 

Total  number  of  student  training  weeks/cycle 
(161.6+6)  -  167.6 

4. 

Average  number  of  student  training  weeks/cycle 
(167.6/20.2)  -  8.3 

5. 

Total  number  of  student  training  weeks /year 
(1032  students  x  8.3  weeks)  -  8566 

Example 

of  Allocation  of  Instructor  Time 

Instructors  spend  37  hours  per  week  on  the  job  (class  is  dismissed 
3  hours  early  on  Friday,  thereby  shortening  the  normal  40  hour  workweek. 
This  works  out  to  296  hours  per  cycle  (8  sequential  weeks/cycle  x  37  hours/ 
week)  spent  by  each  instructor  on  the  job.  Of  these  total  hours,  the 
following  are  devoted  to  non-platform  duties. 

Number  of  Hours 


37  The  average  instructor  is  diverted  completely  from  instructional 
duties  for  a  week  out  of  every  cycle.  During  this  time,  he 
participates  in  a  variety  of  activities  (e.g. ,  lesson  plan  re¬ 
view,  equipment  repair). 
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8  Tuesday  morning  staff  meeting  1  hour 

4  Friday  morning  personnel  inspection  0.5  hour 

5  Student  indoctrination  -  Monday  morning  of  1st  week 

5  Monitoring  student  medical  care  -  Monday  morning  of  7th  week 

7.5  Barracks  cleanup  -  Wednesday  morning  of  1st,  3rd,  and  5th  weeks 
(2.5  hours  each  week) 


66.5  Hours 

66.5  hours  *  296  =  221  =  percentage  of  time  spent  by  average 
instructor  on  non-platform  duties. 

The  rest  of  the  instructor's  time  can  be  analyzed  in  a  similar  fashion. 
This  was  done  for  all  cost  categories.  The  resulting  analysis  is  shown 
below: 


INSTRUCTION/ADMINISTRATIVE  TIME  ALLOCATION 


INSTRUCTORS 

STUDENTS 

Instructor* 

E-6 

(#  lnatruc- 
tore  -  18) 

Sandburg  E-7  OI*on  E-7  Wright  E-7 

Tea*  Supervisor/  Tea*  Supervisor/  Tea*  Supervisor/ 
Instructor  Instructor  Instructor 

Students 

E-2 

Number  of 
students  ■ 

Student  setback* 

r.-2 

Number  of 

1032  students  -  1032 

Parcantaga  of  Salary 
Attributable  to  Coat 
Category  Indlcatad 

69% 

6SZ 

45% 

45% 

15% 

.  bX 

INSTRUCTOR 

TRAINING 

SUPPLIES 

EQUIPMENT 

MAINTENANCE 

lnatruc tore 

E-6 

(#  of  lnatruc- 
tora  ■  16) 

Sandburg  E-7  Olaon  E-7  Wright  E-7 

Tea*  Supervisor/  Tea*  Supervisor/  Tea*  Supervisor 
Instructor  Instructor  Instructor 

Owens  E-0 

Material  Assistant 

Parcantaga  of  Salary 
Attributable  to  Coot 
Category  Indlcatad 
— 

90S 

90% 

90% 

90% 

100% 

ADMINISTRATIVE/ INSTITUTIONAL/ 

miscellaneous 


,  HlehaUi  W-2  Duncan  E-8 

i  Division  "A"  Court* 

'  Director  Coordinator 


Hocktdty  E-7  Cooper  E-7 
Night  lnatruc-  Personal  Affair* 
tor/Sup. rvl#or  Coun.«lor 


Instructors  Data 

E-6  Management 

(f  of  lnatruc-  Croup 
tor#  -  18) 


Percentage  of  Salary  ■ 

Attributable  to  Coat 

Category  Indlcatad  75%  1001  100%  100%  22%  AX 


Sandburg  E-7  Olaon  E-7  Wright  E-7  Naal  E-7 

Tea*  Sugar-  Tea*  Sugar-  Tea*  Super-  Adamletra- 

vieor/ lnatruc-  vtaor/ lnatruc-  vleor/lnatruc-  tlva  Aaalatant 

tor  tor  tor 


Bor  1  E-6 

Admin .  ttrat lva 
Aaalatant 


*«  46%  46%  95% 


95% 


Parcantaga  of  Salar) 
Attributable  to  Coat 
Category  Indlcatad 
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These  time  allocations  were  then  translated  into  billet  and  con^oeite 
military  pay  costs  as  shown  on  pages  D3  and  D4. 
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NAVY  EDUCATION  AND  TRAINING  COST  ANALYSIS 
PROTOTYPE  DATA  COLLECTION  INSTRUMENT 


INSTRUCTOR  SURVEY 


SCHOOL/ SETTING 
NAME  _ 


The  term  "cycle"  will  occur  throughout  this  questionnaire.  Cycle  refers 
to  the  period  of  time  required  to  plan,  design,  implement,  and  evaluate 
a  unit  of  Instruction  within  the  program  being  analyzed.  In  this  ques¬ 
tionnaire,  the  cycle  to  be  considered  is: 


_  .  .  _  _ 

as  the  "cycle  under  discussion." 


hereinafter  described 


The  term  "course"  will  be  used  also.  If  you  do  not  follow  the  traditional 
course  structure,  interpret  the  term  loosely. 


PART  I:  Personnel 


This  portion  of  the  questionnaire  focuses  on  the  amount  of  time  you  spent 
on  the  program  (course /school) . 

1.  You  spend  a  certain  amount  of  time  on  matters  related  exclusively 
to  the  instructional  program.  You  are  also  likely  to  spend  time  on 
matters  related  to  other  aspects  of  Navy  life.  Such  involvement 
might  include  working  on  committees,  advising  trainees,  instructing 
outside  the  program  arena,  conducting  Navy  research,  and  communicating 
with  visitors,  potential  recruits,  and  colleagues. 

Estimate  what  you  believe  represents  your  average  weekly  involvement 
in  these  two  categories  of  activities  during  recent  cycles. 

Instructional  program  activities:  _ hrs/wk 

Other  Navy-related  activities:  _ hrs/wk 

2.  List  below  the  name  and  official  number  of  the  "courses"  in  the 
program  in  which  you  were  involved  and  were  taught  during  the  cycle 
under  discussion.  NOTE:  Not  limited  to  courses  you  taught.  (You 
may  not  require  all  of  the  five  spaces  provided;  conversely,  if 

more  space  is  required,  provide  attachment.)  In  the  last  two  columns, 
indicate  the  number  of  trainees  enrolled  and  how  often  you've  taught 
the  course. 

1. _ _  _  _ 


2. 


3. 


4. 


5. 


The  numbers  1  through  5  above  will  be  used  as  a  shorthand  method  of 
identifying  your  courses  in  the  remaining  portion  of  this  question¬ 
naire. 


EXAMPLE: 


How  many  hours  did  you  spend  this  last  cycle  evaluating  materials 
and/or  the  instructional  process? 

1.  _  2.  _  3.  _  4.  _  5.  _ 

If  the  question  does  not  apply,  enter  "NA"  or  "0." 


NOTE:  If  for  some  reason  you  are  Involved  In,  say,  eight  courses, 
Insert  the  additional  data  In  the  space  below  the  response  line. 

This  question  deals  with  design/planning  functions.  These  activities 
nay  have  occurred  far  in  advance  of  actual  instruction.  They  oust 
have  been  performed  only  for  the  cycle  under  discussion,  i.e.,  if 
you've  taught  a  course  three  times  and  modify  materials  for  each 
cycle,  consider  only  the  modifications  made  for  the  cycle  under  dis¬ 
cussion.  Give  total  hours  per  cycle,  not  hours  per  week.  You  might 
want  to  review  the  following  descriptions  before  you  respond  in  order 
to  avoid  duplicating  your  time  estimate  figures. 

A.  Planning  the  form,  structure  or  content  of  the  program  (not 
individual  courses);  i.e.,  arriving  at  a  general  idea  of  what 
program  will  be: 

TOTAL:  _ 

B.  Planning  form,  structure,  and  general  content  of  courses 
i.e.,  arriving  at  a  general  idea  of  what  course  will  be: 

1.  __ _  2.  _  3.  _ _  4.  _  5.  _ 


C.  Planning  form,  structure,  and  content  of  instructional  materials, 
i.e.,  specific  descriptions  of  course: 

1.  _  2.  _  3.  _ _  4.  _  5.  _ 


D.  Planning  facility  improvements: 

1.  _  2.  _  3.  _  4.  _  5. 


E.  Designing  instructional  materials  after  having  decided  form, 
structure,  and  general  content  (if  you  incorporated  this  data 
in  C,  go  on  to  F) . 

1.  _  2.  _  3.  _  4.  _  5. 


F.  Selecting  instructional  material  already  existing: 

1.  _ _  2.  _  3.  _  4.  _  5. 


G.  Research  to  Increase  personal  knowledge  about  the  program  and 
instruction  in  the  program. 


TOTAL 


H.  Formative  evaluation  of  process,  content,  modules,  or  materials: 

1.  _  2.  _  3.  _  4.  _  5.  _ 

I.  Support  activities  for  any  of  the  above  (include  supervision  of 
support  personnel) : 

1.  _  2.  _  3.  _  4.  _  5.  _ 

This  section  focuses  on  investment/production  functions.  While  the 
activities  may  have  been  performed  far  in  advance  of  instruction, 
they  must  have  been  performed  only  for  the  cycle  under  discussion. 

A.  Purchase  of  hardware  or  equipment: 

1.  _  2.  _  3.  _  4.  _  5.  _ 

B.  Purchase  of  materials  (books,  paper,  etc.,  but  not  office  supplies 
not  used  for  instruction) : 

1.  _  2.  _  3.  _  4.  _  5.  _ 

C.  Purchases  for  renovation  of  facilities  for  specific  instructional 
purposes : 

1.  _  2.  _  3.  _  4.  _  5.  _ 

What  was  the  total  cost  (including  hired  labor)  of  facility 
renovation? 

TOTAL:  $ _ 

D.  Production  of  materials  for  instruction  or  for  trainee  evaluation 
(not  design  of  materials,  but  physical  production  of  them): 

1.  _  2.  _  3.  _  4.  _  5.  _ 

E.  Management  of  personnel  involved  in  the  production  or  purchase 
of  materials  or  equipment: 

1.  _  2.  _  3.  _  4.  _  5.  _ 

F.  Support  activities  for  investment/production  (comparing  prices, 
using  catalogs,  meeting  sales  personnel,  etc.): 


This  section  focuses  on  activities  closely  related  to  Instruction 
during  the  cycle  under  discussion. 

A.  Instruction.  (If  self-instructional  materials  were  used,  there 
may  have  been  no  instruction.  Lab  supervision,  distribution  of 
materials  and  similar  tasks  can  be  accounted  for  in  B): 

1.  _  2.  _  3.  _  4.  . _  5. 


B.  Activities  supporting  instruction  (include  organizational  and 
logistical  activities  for  specific  classes,  but  not  design  of 
materials  for  those  classes) : 

1.  _  2.  _  3.  _  4.  _  5.  _ 

C.  Advising  trainees  in  your  courses 

1.  _  2.  _  3.  _  4.  _  5.  _ 

D.  Advising  other  trainees  enrolled  in  program: 

TOTAL:  ______ 

E.  Trainee  evaluation/assessment: 

1.  _  2.  _  3.  _  4.  _  5.  _ 

F.  Sunmative  evalutlon  of  materials  for  the  instructional  process: 

1.  _  2.  _  3.  _  4.  _  5.  _ 

G.  Dispensing  Information  about  the  program: 

TOTAL:  _ _ 


HL.  Management  and  leadership  connected  with  any  of  the  above  (in¬ 
clude  instructions  to  secretaries,  clerks,  and  assistants): 


PART  II:  Software 


This  part  of  the  questionnaire  attempts  to  identify  the  cost  of  software 
which  was  consumed  for  the  design  or  operation  of  the  cycle  under  dis¬ 
cussion. 

Directions:  The  following  page  provides  three  columns.  Column  one  lists 
classifications  of  softwar-.  In  column  two,  you  are  to  list,  using  the 
number  system  1  through  5  developed  in  PART  I,  by  courses  the  value  of 
the  software  consumed  for  design  activities.  (The  actual  time  that  the 
money  was  spent  is  not  relevant  here.  If  you  use  paper  that  was  purchased 
a  year  ago,  for  example,  treat  is  as  if  you  were  purchasing  it  at  the  time 
of  consumption.)  In  column  three,  perform  the  same  task  for  software 
consumed  during  the  instructional  activity. 

Guideline:  The  value  of  xerox  copies  of  a  chart  produced  for  class  dis¬ 

tribution  should  be  listed  in  column  three.  The  earlier  consumption  of 
material  for  designing  the  chart  (probably  of  very  little  value)  should 
be  listed  in  column  two. 

EXAMPLE: 


Software  Category 


_ Value  of  Software _ 

Design  Instruction 


(1) 


(2) 

(design) 


(3) 

(instruction) 


PAPER 

PRINTING 

FILM 

FILMSTRIP 
SLIDES 
STILL  PHOTOS 

TRANSPARENCIES/VIEWGRAPHS 
VIDEO  TAPE 


AUDIO  TAPE 


PART  III:  Equipment  and  Hardware 


In  this  part  of  the  questionnaire,  you  are  asked  to  identify  the  number 
of  hours  that  equipment  and  hardware  was  used  in  order  to  implement 
your  courses  during  the  cycle  under  discussion. 

1.  Use  the  checklist  below  to  identify  hardware  and  equipment  used  for 
the  design,  production,  or  implementation  of  your  course. 

HARDWARE 


Cameras : 

_TV 

_16mm 

_8mm 

Projectors: 
_Overhead 
jOpaque 
_Slide 
_Filmstrip 
_16mm  film 
_8mm  film  (reel) 
_8mm  (filmloop) 


_Screens 

_TV  Monitors  (Viewing) 
_Tape  Recorders  (reel) 
_Cassette  Recorders 
_Record  Players 
_Teaching  Machines 
(identify  type: 


_ _ _ ) 

Microphones 

Headsets 


EQUIPMENT 

(if  more  than  one,  identify  specifically) 


-Typewriter  _Xerox 

- - -  _Other  duplicating  equip- 

- —  ment  (specify  type) 


_Calculator 

Thermofax 

Ditto 

Offset 
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•t 


2 


The  table  on  the  following  page  ehould  be  filled  out  as  follows: 


Column  I:  Name  the  piece  of  equipment  or  hardware.  BE  SPECIFIC. 
Provide  information  that  will  allow  Investigators  to  estimate 
value. 

Column  II:  Identify  source  Cexample:  "dept.."  "media  center," 
"ray  own") . 

Columns  III  and  IV:  Enter  the  number  of  hours  the  equipment  was 
used  according  to  the  nature  of  the  task  (Des.  =  design,  and 
Inst.  -  instruction).  Provide  estimates  by  individual  courses, 
using  the  number  system  1  through  5. 

Column  V:  Check  if  you  used  this  equipment  or  hardware  outside 
of  the  program  being  analyzed. 


EXAMPLE: 


(I) 

NAME/ ID 

(ID 

Source 

(Ill) 
Hrs.  used 

Des. 

(IV) 

this  cycle 

Inst. 

(V) 

Non-Program 

Cassette  Sony  0120 

own 

2=5 

— 

16mm  Projector 

Media 

Center 

— 

1=1 

3=2 

X 

Xerox  4000 

Inst. 

Div. 

1-1/2 

1=1/2 

2-1/2 

3-1/2  X 


Notes  concerning  Xerox  Example:  Design  Time  -  Rough  drafts;  InBt  Time  - 
Class  handouts. 
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PART  IV:  Services 


The  following  page  contains  a  sample  completed  matrix  Identifying  possible 
instructional  services,  how  they  were  "paid  for,"  and  utilized.  The  next 
page  is  a  matrix  for  you  to  use  in  identifying  any  services  used  in  the 
cycle  being  analyzed. 
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PART  V:  Support  Personnel 


In  this  final  part  of  the  survey,  you  are  asked  to  identify  all  support 
personnel  who  have  worked  for  you  on  the  program  and  indicate  the  number 
of  hours  they  worked.  Support  personnel  are  secretaries,  assistants, 
and  other  helpers,  paid  by  the  unit  administering  the  program,  who 
assisted  you  in  some  manner.  Also  identify  trainees  who  also  served 
in  an  assistant /aid  capacity. 
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TOTAL  HOURS  WORKED  FOR  YOU  THIS  CYCLE 


V.  SUPPORT  PERSONNEL 


THIS  IS  THE  LAST  PART  OF  OUR  SURVEY. 
THANK  YOU  SO  MUCH  FOR  COOPERATING! 


NAVY  EDUCATION  AND  TRAINING  COST  ANALYSIS 
PROTOTYPE  DATA  COLLECTION  INSTRUMENT 

ADMINISTRATOR  SURVEY  NAME  _ 

The  activities  on  which  this  questionnaire  will  focus  are  those 
associated  with  the  program/course/cycle. 


1.  Estimate  what  you  believe  represents  your  average  total  weekly 
involvement  as  an  administrator.  Include  all  activities  which 
relate  specifically  to  your  position  as  an  administrator: 

_  hrs/wk 

The  following  questions  refer  only  to  your  involvement  in  the 
identified  program. 

2.  Identify  the  total  amount  of  time  spent  as  an  administrator, 

planning  and  designing  the  program  relative  to  the  _ 

cycle : 

_  total  hrs/cycle 

3.  Identify  the  total  amount  of  time  spent  which  waB  most  closely 
related  to  purchases  for  the  program: 

_  total  hrs/cycle 

4.  Identify  the  total  amount  of  time  spent  on  matters  in  support  of 

the  execution  of  the  _  cycle: 

_ total  hrs/cycle 

5.  Estimate  the  cost  of  materials  consumed  in  support  of  the  administra¬ 
tive  activities  described  in  Items  2,  3,  and  4  above. 

$ _ 

6.  Identify  any  consultant  retained  for  the  program.  List  fee  and 
expenses  and  briefly  describe  his  contribution. 


7.  Identify  the  extent  to  which  you  used  institutional  resources  for 
administration  of  the  program  by  completing  the  following  page 
(page  three  ie  an  example  of  how  the  form  may  be  used). 
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DISTRIBUTION  LIST 


Assistant  Secretary  of  the  Navy  (Manpower  and  Reserve  Affairs) 

Chief  of  Naval  Operations  (OP-98 7P10) 

Chief  of  Naval  Operations  (OP-99)  (3) 

Chief  of  Naval  Personnel  (Pers-lOc) 

Chief  of  Naval  Material  (NMAT  0344) 

Chief  of  Naval  Material  (NMAT  035) 

Chief  of  Naval  Research  (Code  450)  (4) 

Chief  of  Naval  Education  and  Training  (N5) 

Chief  of  Naval  Education  and  Training  (005) 

Chief  of  Naval  Education  and  Training  (00A) 

Chief  of  Naval  Education  and  Training  (N6) 

Chief  of  Naval  Education  and  Training  Support 
Chief  of  Naval  Education  and  Training  Support  (01A) 

Chief  Naval  Technical  Training  (Code  016) 

Commander  Training  Command,  U.S.  Pacific  Fleet 
Commander  Training  Command,  U.S.  Atlantic  Fleet  (Code  N3A) 

Commanding  Officer,  Fleet  Combat  Direction  Systems  Training  Center, 

Pacific  (Code  00E) 

Commanding  Officer,  Fleet  Training  Center,  San  Diego 

Commanding  Officer,  Naval  Education  and  Training  Program  Development  Center 
Commanding  Officer,  Naval  Development  and  Training  Center  (Code  0120) 
Director,  Training  Analysis  and  Evaluation  Group  (TAEG) 

Personnel  Research  Division,  Air  Force  Human  Resources  Laboratory  (AFSC) , 
Lackland  Air  Force  Base 

Technical  Training  Division,  Air  Force  Human  Resources  Laboratory, 

Lowry  Air  Force  Base 

Flying  Training  Division,  Air  Force  Human  Resources  Laboratory, 

Williams  Air  Force  Base 

Technical  Library,  Air  Force  Human  Resources  Laboratory, 

Lackland  Air  Force  Base 


